Hi Luke,
Thank you for your reply.
1) How big are your packets?
They are 64 bytes long of payload.
2) Are you using normal network sockets, or something like netmap?
Normal network raw sockets.
3) What hardware and software are you using on the host you send packets from?
Both hosts are the same:
HW - recent Intel New Unit Computing device, i5-4250U.
SW - Fedora core 19, kernel 3.14.
4) Do you really always wait for the reception of an echo packet
before you send the next one?
Here is how it works:
Let's assume for example I'll be waiting 500 usec between sending packets:
Send a packet, wait for the packet to be echoed back OR 500 usec expired
and send the next packet.
I get around 110 usec averaged rtt between the time I send, to the time I
receive the echoed packet back. The best case rtt is around 80 usec.
If I use 70 usec instead as the interval between sending packets, I get an
average of 80 usec and best case rtt is around 50 usec.
BTW, in both cases, I see that the first rtt is around 1.5 msec.
The above results tell me that the HW(?) is holding back before generating
the interrupts.
I traced e1000e kernel functions and the actual RX and TX processing is
fast enough. So it does not come from this part.
Thank you,
Kobi
Hi,
note that I've never used e1000e, but based on my experience with ixgbe:
if you use e1000e in NAPI (RX_POLLING) mode then Linux kernel will
queue net_rx_action softirq task and disable RX Interrupts until
net_rx_action completes. So, in effect, if you're using the NAPI,
there will always be an interrupt moderation effect happening at high
enough network loads.
However, I can't really match your experience to anything I would
expect the driver or NAPI to be doing. From what the e1000e readme[0]
tells me, its Interrupt Throttling seems to be following the same idea
as in ixgbe and it appears to be using the same ITR hardware feature.
The ITR hardware feature is a "cooldown" timer that starts when an rx
interrupt is asserted and blocks the assertion of the rx interrupt
until the timer has expired. If there is a packet in the rx queue when
the timer expires, an interrupt is asserted immediately. Otherwise, it
will be asserted the next time a packet is received.
However, there is nothing in the ITR or in the NAPI that would cause
*lower* latency as the network load *rises*, except when the NAPI is
constantly in polling mode because net_rx_action never manages to
empty the rx queue, but this only consistently happens when the
network load is extremely stable because it is a pretty instable
state.
Maybe this gives you some more context to think about what is
happening in your network - I can't really point at anything specific.
Here are some questions:
1) How big are your packets?
2) Are you using normal network sockets, or something like netmap?
3) What hardware and software are you using on the host you send packets from?
4) Do you really always wait for the reception of an echo packet
before you send the next one?
In general, user space network sockets will give you terrible
performance compared to what is possible if you bypass the linux
network system. You might want to look at dpdk and netmap, or PF_RING
ZC.
I don't know if e1000e supports it, but there is also LLDP / busy read
sockets to coax some more performance out of linux network sockets.
[1]
Best,
Luke
[0] http://downloadmirror.intel.com/20927/eng/e1000.htm#e1000e
[1] http://lwn.net/Articles/551284
- Kobi
Like "Pike-It Pilates Studio" on Facebook:
Pike It - Pilates Studio
<https://www.facebook.com/pages/Pike-It-Pilates-Studio/1583096505242821?sk=photos_stream&ref=page_internal>
Please support me in Pedal The Cause
<http://sandiego.pedalthecause.org/riders_profile.jsp?MemberID=6747> - "A
world without cancer"
__o
_ `\ <,_
(*) / (*)
On Mon, Apr 20, 2015 at 10:39 PM, Kobi Cohen-Arazi <
kobi.cohenar...@gmail.com> wrote:
> Hi All,
>
> I'm using FC19, kernel 3.14, e1000e driver with I218-v NIC.
>
> I'm trying to disable any RX interrupt mitigation/throttling/coalescing.
>
> I tried using InterruptThrottleRate=0, RxIntDelay=0, SmartPowerDownEnable=0,
> rx-usecs=0,
> but I still see the driver throttle RX interrupts. My application requires
> low latency and I'd like to have RX interrupt as soon as packet is received.
>
> I have a simple test app that compute RTT between 2 nodes.
> All it does is to send a packet to a remote node, wait for this packet to
> be echoed back and measures the rtt.
> I can see that the longer I wait between 2 packets, the longer the rtt is
> (until a certain point where it is stabilized). If I shorten the rtt and
> send packets faster (send, wait for echo reply, send the next packet) then
> the rtt is shorter. There is definitely some kind of interrupt mitigation
> and the driver tries to bundle interrupts/recv packets together.
>
> Besides the above modules params and ethtool coalesce param (rx-usecs) is
> there anything else needed in order to completely disable any coalescing /
> RX interrupt mitigation?
>
> Thank you in advance,
> - Kobi
>
> __o
> _ `\ <,_
> (*) / (*)
>
------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
