On Wed, Aug 29, 2018 at 12:42:33PM +0300, Paul wrote: > Hello team, > > > It seems like a commit on Mar 23 introduced a bug: if during execution of > arc_adjust() > target is reached after MRU is evicted current code continues evicting MFU. > Before said > commit, on the step prior to MFU eviction, target value was recalculated as: > > target = arc_size - arc_c; > > arc_size here is a global variable that was being updated accordingly, during > MRU eviction, > hence this expression, resulted in zero or negative target if MRU eviction > was enough > to reach the original goal. > > Modern version uses cached value of arc_size, called asize: > > target = asize - arc_c; > > Because asize stays constant during execution of whole body of arc_adjust() > it means that > above expression will always be evaluated to value > 0, causing MFU to be > evicted every > time, even if MRU eviction has reached the goal already. Because of the > difference in > nature of MFU and MRU, globally it leads to eventual reduction of amount of > MFU in ARC > to dramatic numbers.
Hi Paul, Your analysis does seem right to me. I cc'ed the openzfs mailing list so that an actual ZFS expert can chime in; it looks like this behaviour is consistent between FreeBSD, illumos and ZoL. Have you already tried the obvious "fix" of subtracting total_evicted from the MFU target? > Servers that run the version of FreeBSD prior to the issue have this picture > of ARC: > > ARC: 369G Total, 245G MFU, 97G MRU, 36M Anon, 3599M Header, 24G Other > > As you can see, MFU dominates. This is a nature of our workload: we have a > considerably > small dataset that we use constantly and repeatedly; and a large dataset that > we use > rarely. > > But on the modern version of FreeBSD picture is dramatically different: > > ARC: 360G Total, 50G MFU, 272G MRU, 211M Anon, 7108M Header, 30G Other > > This leads to a much heavier burden on the disk sub-system. > > > Commit that introduced a bug: > https://github.com/freebsd/freebsd/commit/555f9563c9dc217341d4bb5129f5d233cf1f92b8 ------------------------------------------ openzfs: openzfs-developer Permalink: https://openzfs.topicbox.com/groups/developer/T10a105c53bcce15c-M8152dc2430a5ea4e625ad564 Delivery options: https://openzfs.topicbox.com/groups/developer/subscription
