No, in this case it is talking about the offsets of the entries. When we do GC, we read from the old (partially empty) log and copy any non-deleted entries we find into the new entry log. This means that the index location for that entry needs to change to point to the new entrylog and new offset. We cannot update the index though, until the new entry log has flushed. Otherwise, we could update the index, then crash, the new index entry would point to memory which had never been flushed. We cannot flush the entry log for each entry, because that would slow everything down. So we batch. We copy over 'maxOutstandingRequests' entries, keep a record of the new offsets, and then wait for a flush of the entrylog, or trigger one (I can't remember which). Once the flush has occurred, the offsets can be added to the index.
Let us know if this isn't clear, as it most likely isn't :) -Ivan On Sat, Dec 13, 2014 at 4:12 PM, [email protected] <[email protected]> wrote: > Hi, > > I am preparing a patch for > https://issues.apache.org/jira/browse/BOOKKEEPER-827 , in which try to > turn maxOutstandingRequests from "by entries" to "by bytes". > while reading the part of comments at the front of > setCompactionMaxOutstandingRequests(), it makes me a little confusing: > "A higher value for this parameter means more memory will be used for > offsets" > <=== here, I thought the memory should mainly be occupied by the entries > that added into "FileChannel", but not flushed. right? > > bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java > : line 1231 > ------------------------ > /** > * Set the maximum number of entries which can be compacted without > flushing. > * > * When compacting, the entries are written to the entrylog and the > new offsets > * are cached in memory. Once the entrylog is flushed the index is > updated with > * the new offsets. This parameter controls the number of entries > added to the > * entrylog before a flush is forced. A higher value for this > parameter means > * more memory will be used for offsets. Each offset consists of 3 > longs. > * > * This parameter should _not_ be modified unless you know what you're > doing. > * The default is 100,000. > * > * @param maxOutstandingRequests number of entries to compact before > flushing > * > * @return ServerConfiguration > */ > public ServerConfiguration setCompactionMaxOutstandingRequests(int > maxOutstandingRequests) { > setProperty(COMPACTION_MAX_OUTSTANDING_REQUESTS, > maxOutstandingRequests); > return this; > } > ------------------------ > > Thanks a lot. > -Jia >
