|
http://deinoscloud.wordpress.com/2009/06/29/virtual-interrupt-coalescing/ The performance of high-IOPS workloads can be improved, in terms of higher throughput and lower CPU utilization, by turning on interrupt coalescing for Fibre Channel HBAs. This optimization has been in place since ESX 3.5. Analogous to this, in order to further reduce the CPU cost of I/O in high-IOPS workloads, ESX 4.0 implements virtual interrupt coalescing to allow for some batching of I/O completions to happen at the guest level. In order to avoid the undesirable side-effect of increasing latency, this coalescing only occurs when I/O levels exceed a threshold. This threshold is set high enough that performance does not degrade under trickle I/O or latency-bound conditions. Another optimization, related to inter-processor interrupt (IPI) coalescing, addresses a problem specific to virtual environments. Hardware interrupts that signal I/O completions are routed by the ESX storage stack to the virtual machine that issued the I/O. If the interrupt is not received on the physical CPU where the relevant VMM is executing, an IPI must be sent to the pCPU executing that VMM. As processor core density increases, it becomes more likely that hardware interrupts will be received on processors not running the target virtual machine, thus increasing the number of times the IPIs need to be issued. In ESX 4.0, we provide a mechanism to reduce the number of IPIs issued using the time-stamp of the last interrupt that was sent to the guest operating system. Before sending an IPI, we check the current time against this timestamp. If the time difference is greater than a threshold, 100μs by default, the IPI is posted. Otherwise, the IPI is delayed and the VMM has the opportunity to batch it with other I/O completions at a later time. This reduces the overall number of IPIs while bounding the latency of notifying the guest operating system about an I/O completion. As with other interrupts, lowering the rate of IPIs reduces the CPU cost for this benchmark. For a detailed discussion of interrupt coalescing improvements, see the following publication: Irfan Ahmad, Ajay Gulati, and Ali Mashtizadeh, “Improving Performance with Interrupt Coalescing for Virtual Machine Disk I/O in VMware ESX Server,” Second International Workshop on Virtualization Performance: Analysis, Characterization, and Tools (VPACT’09), held with IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2009. |
