Matt, Sephe, Thank you for the prompt and detailed feedback. I will take a look at the ipitest script, and the dedicated IPI vector path. I am interested in the overall latencies of IPIs themselves, and whether (at some point) they'd be useful in my research to trigger from user-level for certain purposes.
-Alex On Wed, Aug 3, 2016 at 1:56 AM, Matthew Dillon <dil...@backplane.com> wrote: > The main improvement was to the MMU page-invalidation routines. In 4.4 > these routines used the kernel's generic IPI messaging API. But page > invalidations require all cpus to synchronize their changes and since the > generic IPI messaging API honors critical sections that could cause most of > the cpu's to stall for longer than intended if just one of the cpus > happened to be in a long critical section. In 4.6 the page-invalidation > code runs through a dedicated IPI vector which ignores critical sections > and does not have this stall issue. > > A second optimization as also implemented, but could not be tested well > enough to enable for the release. This optimization can be turned on with > a sysctl in the 4.6 release (sysctl machdep.optimized_invltlb=1). This > optimization is meant to avoid sending IPIs to CPUs which are idle and thus > might be in a low-power mode. Such cpus will respond much more slowly to > the IPI vector and not only increase latency for the cpus running at full > power, but also have a tendancy to kick the cpus in low-power mode out of > low-power mode. This second optimization is most useful on systems with a > lot of cores (multi-socket systems and systems with > 4 cores). This one > will eventually be turned on by default once sufficient testing of the > feature has been done. > > There were numerous other optimizations to reduce the amount of IPI > signalling needed. Probably the biggest one is that the buffer cache no > longer synchronizes the MMU when throwing away a buffer cache buffer. It > only synchronizes the MMU when allocating one. This cut out roughly half > of the system-wide IPIs in a nominally loaded system. We have an > additional optimization, also disabled by default, which nearly eliminates > buffer-cache-related IPIs in situations where filesystem disk bandwidth is > very high (sysctl vfs.repurpose_enable=1). It's only really applicable > under heavy loads when consolidated disk I/O bandwidth exceeds 200 > MBytes/sec. > > -Matt > >
