Adaptive IRQ moderation (also called adaptive IRQ coalescing) has been widely
used
in the networking stack for over 20 years and has become a standard default
setting.
Adaptive moderation is a feature supported by the device to delay an interrupt
for a either a period of time, or number of completions (packets for networking
devices) in order to optimize the effective throughput by mitigating the cost of
handling interrupts which can be expensive in modern rates in networking and/or
storage devices.
The basic concept of adaptive moderation is to provide time and packet based
control
of when the device generates an interrupt in an adaptive fashion, attempting to
identify the current workload based on online gathered statistics instead of a
predefined configuration. When done correctly, adaptive moderation can win the
best of both throughput
This rfc patchset introduces a generic library that provides the mechanics for
online statistics monitoring and decision making for the consumer which simply
need to program the actual device while still keeping room for device specific
tuning.
In the networking stack, each device driver implements adaptive IRQ moderation
on its own. The approach here is a bit different, it tries to take the common
denominator,
which is per-queue statistics gathering and workload change identification
(basically decides if the moderation scheme needs to change).
The library is targeted to multi-queue devices, but should work on single queue
devices as well, however I'm not sure that these devices will need something
like interrupt moderation.
The model used in the proposed implementation requires the consumer (a.k.a the
device driver) to initialize an irq adaptive moderator context (a.k.a irq_am)
per
its desired context which will most likely be a completion queue.
The moderator is initialized with a set of am (adaptive moderation) levels,
which are essentially the abstraction of the device specific moderation
parameters.
Usually the different levels will map to different pairs of
(time <usecs>, completion-count) which are left specific to the consumer.
The moderator assumes that the am levels are sorted in an increasing order when
the
lowest level corresponds to the optimum latency tuning (short time and low
completion-count) and gradually increasing towards the throughput optimum tuning
(longer time and higher completion-count). So there is a trend and tuning
direction
tracked by the moderator. When the moderator collects sufficient statistics
(also
controlled by the consumer defining nr_events), it compares the current stats
with the
previous stats and if a significant changed was observed in the load, the
moderator
attempts to increment/decrement its current level (called a step) and schedules
a program
dispatch work.
The main reason why this implementation is different then the common networking
devices
implementation (and kept separate) is that in my mind at least, network devices
are different
animals than other I/O devices in the sense that:
(a) network devices rely heavily on byte count of raw ethernet frames for
adaptive moderation
while in storage or I/O, the byte count is often a result of a
submission/completion transaction
and sometimes even known only to the application on top of the
infrastructure (like in the
rdma case).
(b) Performance characteristics and expectations in representative workloads.
(c) network devices collect all sort of stats for different functionalities
(where adaptive moderation
is only one use-case) and I'm not sure at all that a subset of stats could
easily migrate to a different
context.
Having said that, if sufficient reasoning comes along, I could be convinced
otherwise if unification
of the implementations between networking and I/O is desired.
Additionally, I hooked two consumers into this framework:
1. irq-poll - interrupt polling library (used by various rdma consumers and can
be extended to others)
2. rdma cq - generic rdma cq polling abstraction library
With this, both RDMA initiator mode consumers and RDMA target mode consumers
are able to utilize the framework (nvme, iser, srp, nfs). Moreover, I currently
do
not see any reason why other HBAs (or devices in general) that support interrupt
moderation wouldn't be able to hook into this framework as well.
Note that the code is in *RFC* level, attempting to convey the concept.
If the direction is acceptable, the setup can be easily modified and
cleanups can be performed.
Initial benchmarking shows promising results with 50% improvement in throughput
when testing high load of small I/O. The experiment taken used nvme-rdma host
vs. nvmet-rdma
target exposing a null_blk device. The workload ran multithreaded fio run with
high queue-depth
and block size of 512B read I/O (4K block size would exceed 100 Gbe wire speed).
The results without adaptive moderation reaches ~8M IOPs bottlenecking the host
and target cpu.
With adaptive moderation enabled, IOPs quickly converge to ~12M IOPs (at the
expense of slightly
higher latencies obviously) and debugfs stats show that the moderation level
reached the
throughput optimum level. There is currently a known issue I've observed in
some conditions
converging back to latency optimum (after reaching throughput optimum am
levels) and I'll work
to fix the tuning algorithm. Thanks to Idan Burstein for running some
benchmarks on his
performance setup.
I've also tested this locally with my single core VMs and saw similar
improvement of ~50%
in throughput in a similar workload (355 KIOPs vs. 235 KIOPs). More testing
will help
a lot to confirm and improve the implementation.
QD=1 Latency tests showed a marginal regression of up to 2% in latency (lightly
tested though).
The reason at this point is that the moderator still bounces in the low latency
am levels
constantly (would like to improve that).
Another observed issue is the presence of user context polling (IOCB_HIPRI)
which does not update
the irq_am stats (mainly because its not interrupt driven). This can cause the
moderator to do
the wrong thing as its based on partial view of the load (optimize for latency
instead of getting
out of the poller's way). However, recent discussions raised the possibility
that polling requests
will be executed on a different set of queues with interrupts disabled
altogether, which would make
this a non-issue.
None the less, I would like to get some initial feedback on the approach. Also,
I'm not an expert
in tuning the algorithm. The basic approach was inspired by the mlx5 driver
implementation which
seemed the closest to fit the abstraction level that I was aiming for. So I'd
also love to get some
ideas on how to tune the algorithm better for various workloads (hence the RFC).
Lastly, I have also attempted to hook this into nvme (pcie), but that wasn't
successful mainly
because the coalescing set_feature is global to the controller and not
per-queue.
I'll be looking to bringing per-queue coalescing to the NVMe TWG (in case the
community is
interested in supporting this).
Feedback would be highly appreciated, as well as a test drive with the code in
case anyone
is interested :)
Sagi Grimberg (5):
irq-am: Introduce helper library for adaptive moderation
implementation
irq-am: add some debugfs exposure on tuning state
irq_poll: wire up irq_am and allow to initialize it
IB/cq: add adaptive moderation support
IB/cq: wire up adaptive moderation to workqueue based completion
queues
drivers/infiniband/core/cq.c | 73 ++++++++++-
include/linux/irq-am.h | 118 ++++++++++++++++++
include/linux/irq_poll.h | 9 ++
include/rdma/ib_verbs.h | 9 +-
lib/Kconfig | 6 +
lib/Makefile | 1 +
lib/irq-am.c | 291 +++++++++++++++++++++++++++++++++++++++++++
lib/irq_poll.c | 30 ++++-
8 files changed, 529 insertions(+), 8 deletions(-)
create mode 100644 include/linux/irq-am.h
create mode 100644 lib/irq-am.c
--
2.14.1
