I'll reply on the JIRA. On Tue, Dec 16, 2014 at 6:51 AM, Jia Zhai <[email protected]> wrote:
> Hi Ivan, > > Thanks a ton for your explain. It is very clear. > > If you have time, Would you please help take a look at bookkeeper-827? > https://issues.apache.org/jira/browse/BOOKKEEPER-827 > > After this enhancement, user could choose to use "compactionRate" > and "maxOutstandingRequests" either "by entries" or "by bytes". And new > parameter "maxOutstandingRequestsBytes" stands for number of bytes that > entries have occupied before flush. > > > Thanks a lot. > > -Jia > > On Tue, Dec 16, 2014 at 12:14 AM, Ivan Kelly <[email protected]> wrote: > > > > No, in this case it is talking about the offsets of the entries. > > > > When we do GC, we read from the old (partially empty) log and copy any > > non-deleted entries we find into the new entry log. This means that the > > index location for that entry needs to change to point to the new > entrylog > > and new offset. We cannot update the index though, until the new entry > log > > has flushed. Otherwise, we could update the index, then crash, the new > > index entry would point to memory which had never been flushed. We cannot > > flush the entry log for each entry, because that would slow everything > > down. So we batch. We copy over 'maxOutstandingRequests' entries, keep a > > record of the new offsets, and then wait for a flush of the entrylog, or > > trigger one (I can't remember which). Once the flush has occurred, the > > offsets can be added to the index. > > > > Let us know if this isn't clear, as it most likely isn't :) > > > > -Ivan > > > > On Sat, Dec 13, 2014 at 4:12 PM, [email protected] < > [email protected]> > > wrote: > > > > > Hi, > > > > > > I am preparing a patch for > > > https://issues.apache.org/jira/browse/BOOKKEEPER-827 , in which try to > > > turn maxOutstandingRequests from "by entries" to "by bytes". > > > while reading the part of comments at the front of > > > setCompactionMaxOutstandingRequests(), it makes me a little confusing: > > > "A higher value for this parameter means more memory will be used for > > > offsets" > > > <=== here, I thought the memory should mainly be occupied by the > entries > > > that added into "FileChannel", but not flushed. right? > > > > > > > > > bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java > > > : line 1231 > > > ------------------------ > > > /** > > > * Set the maximum number of entries which can be compacted without > > > flushing. > > > * > > > * When compacting, the entries are written to the entrylog and the > > > new offsets > > > * are cached in memory. Once the entrylog is flushed the index is > > > updated with > > > * the new offsets. This parameter controls the number of entries > > > added to the > > > * entrylog before a flush is forced. A higher value for this > > > parameter means > > > * more memory will be used for offsets. Each offset consists of 3 > > > longs. > > > * > > > * This parameter should _not_ be modified unless you know what > > you're > > > doing. > > > * The default is 100,000. > > > * > > > * @param maxOutstandingRequests number of entries to compact > before > > > flushing > > > * > > > * @return ServerConfiguration > > > */ > > > public ServerConfiguration setCompactionMaxOutstandingRequests(int > > > maxOutstandingRequests) { > > > setProperty(COMPACTION_MAX_OUTSTANDING_REQUESTS, > > > maxOutstandingRequests); > > > return this; > > > } > > > ------------------------ > > > > > > Thanks a lot. > > > -Jia > > > > > >
