W dniu 2017-10-19 o 01:37, Alexander Duyck pisze:
On Wed, Oct 18, 2017 at 4:22 PM, Paweł Staszewski <pstaszew...@itcare.pl> wrote:
W dniu 2017-10-19 o 00:58, Paweł Staszewski pisze:
W dniu 2017-10-19 o 00:50, Paweł Staszewski pisze:
W dniu 2017-10-19 o 00:20, Paweł Staszewski pisze:
W dniu 2017-10-18 o 17:44, Paweł Staszewski pisze:
W dniu 2017-10-17 o 16:08, Paweł Staszewski pisze:
W dniu 2017-10-17 o 13:52, Paweł Staszewski pisze:
W dniu 2017-10-17 o 13:05, Paweł Staszewski pisze:
W dniu 2017-10-17 o 12:59, Paweł Staszewski pisze:
W dniu 2017-10-17 o 12:51, Paweł Staszewski pisze:
W dniu 2017-10-17 o 12:20, Paweł Staszewski pisze:
W dniu 2017-10-17 o 11:48, Paweł Staszewski pisze:
W dniu 2017-10-17 o 02:44, Paweł Staszewski pisze:
W dniu 2017-10-17 o 01:56, Alexander Duyck pisze:
On Mon, Oct 16, 2017 at 4:34 PM, Paweł Staszewski
<pstaszew...@itcare.pl> wrote:
W dniu 2017-10-16 o 18:26, Paweł Staszewski pisze:
W dniu 2017-10-16 o 13:20, Pavlos Parissis pisze:
On 15/10/2017 02:58 πμ, Alexander Duyck wrote:
Hi Pawel,
To clarify is that Dave Miller's tree or Linus's that you
are talking
about? If it is Dave's tree how long ago was it you pulled
it since I
think the fix was just pushed by Jeff Kirsher a few days
ago.
The issue should be fixed in the following commit:
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/drivers/net/ethernet/intel/i40e/i40e_txrx.c?id=2b9478ffc550f17c6cd8c69057234e91150f5972
Do you know when it is going to be available on net-next
and linux-stable
repos?
Cheers,
Pavlos
I will make some tests today night with "net" git tree where
this patch is
included.
Starting from 0:00 CET
:)
Upgraded and looks like problem is not solved with that patch
Currently running system with
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/
kernel
Still about 0.5GB of memory is leaking somewhere
Also can confirm that the latest kernel where memory is not
leaking (with
use i40e driver intel 710 cards) is 4.11.12
With kernel 4.11.12 - after hour no change in memory usage.
also checked that with ixgbe instead of i40e with same
net.git kernel there
is no memleak - after hour same memory usage - so for 100%
this is i40e
driver problem.
So how long was the run to get the .5GB of memory leaking?
1 hour
Also is there any chance of you being able to bisect to
determine
where the memory leak was introduced since as you pointed out
it
didn't exist in 4.11.12 so odds are it was introduced
somewhere
between 4.11 and the latest kernel release.
Can be hard cause currently need to back to 4.11.12 - this is
production host/router
Will try to find some free/test router for tests/bicects with
i40e driver (intel 710 cards)
Thanks.
- Alex
Also forgoto to add errors for i40e when driver initialize:
[ 15.760569] i40e 0000:02:00.1: Error I40E_AQ_RC_ENOSPC adding
RX filters on PF, promiscuous mode forced on
[ 16.365587] i40e 0000:03:00.3: Error I40E_AQ_RC_ENOSPC adding
RX filters on PF, promiscuous mode forced on
[ 16.367686] i40e 0000:02:00.2: Error I40E_AQ_RC_ENOSPC adding
RX filters on PF, promiscuous mode forced on
[ 16.368816] i40e 0000:03:00.0: Error I40E_AQ_RC_ENOSPC adding
RX filters on PF, promiscuous mode forced on
[ 16.369877] i40e 0000:03:00.2: Error I40E_AQ_RC_ENOSPC adding
RX filters on PF, promiscuous mode forced on
[ 16.370941] i40e 0000:02:00.3: Error I40E_AQ_RC_ENOSPC adding
RX filters on PF, promiscuous mode forced on
[ 16.372005] i40e 0000:02:00.0: Error I40E_AQ_RC_ENOSPC adding
RX filters on PF, promiscuous mode forced on
[ 16.373029] i40e 0000:03:00.1: Error I40E_AQ_RC_ENOSPC adding
RX filters on PF, promiscuous mode forced on
some params that are set for this nic's
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
ethtool -G $i rx 1024 tx 2048
ip link set $i txqueuelen 1000
ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
512 tx-usecs 128
ethtool -L $i combined 6
#ethtool -N $i rx-flow-hash udp4 sdfn
ethtool -K $i ntuple on
ethtool -K $i gro off
ethtool -K $i tso off
Also after TSO/GRO on there is memory usage change - and leaking
faster
Below image from memory usage before change with TSO/GRO OFF and
after enabling TSO/GRO
https://ibb.co/dTqBY6
Thanks
Pawel
With settings like this:
ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
enp3s0f2 enp3s0f3'
for i in $ifc
do
ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 512
tx-usecs 128
ethtool -K $i gro on
ethtool -K $i tso on
done
Server is leaking about 4-6MB per each 10 seconds
MEMLEAK:
5 MB/10sec
6 MB/10sec
4 MB/10sec
4 MB/10sec
Other settings TSO/GRO off
ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
enp3s0f2 enp3s0f3'
for i in $ifc
do
ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 512
tx-usecs 128
ethtool -K $i gro off
ethtool -K $i tso off
done
Same leak about 5MB per 10 seconds
MEMLEAK:
5 MB/10sec
5 MB/10sec
5 MB/10sec
Other settings rx-usecs change from 512 to 1024:
ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1
enp3s0f2 enp3s0f3'
for i in $ifc
do
ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs
1024 tx-usecs 128
ethtool -K $i gro off
ethtool -K $i tso off
done
MEMLEAK:
4 MB/10sec
3 MB/10sec
4 MB/10sec
4 MB/10sec
So memleak have something to do with rx-usecs (less interrupts but
bigger latency for traffic)
But also enabling TSO/GRO making leak about 1MB bigger for each 10
seconds
So far best config is:
ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
enp3s0f3'
for i in $ifc
do
ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 64
tx-usecs 512
ethtool -K $i gro off
ethtool -K $i tso on
done
MEMLEAK - about 2MB/10secs
2 MB/10sec
2 MB/10sec
2 MB/10sec
With - rx-usecs set to 256 (about 7-9MB/10secs memleak)
ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
enp3s0f3'
for i in $ifc
do
ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 256
tx-usecs 512
ethtool -K $i gro off
ethtool -K $i tso on
done
MEMLEAK:
7 MB/10sec
7 MB/10sec
8 MB/10sec
9 MB/10sec
And even less memleak with rx-usecs set to 32
ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
enp3s0f3'
for i in $ifc
do
ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 32
tx-usecs 512
ethtool -K $i gro off
ethtool -K $i tso on
done
MEMLEAK - about 0-2MB for each 10 seconds
0 MB/10sec
1 MB/10sec
0 MB/10sec
2 MB/10sec
1 MB/10sec
So best settings - to have as less leak as possible for now (rx-usecs
set to 16):
ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
enp3s0f3'
for i in $ifc
do
ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 16
tx-usecs 768
ethtool -K $i gro on
ethtool -K $i tso on
done
MEMLEAK: (0-1MB/10seconds)
0 MB/10sec
0 MB/10sec
0 MB/10sec
1 MB/10sec
1 MB/10sec
-1 MB/10sec
1 MB/10sec
1 MB/10sec
0 MB/10sec
(there are some memory recycles - so this is good :) )
Compared to(rx-usecs 512):
ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
enp3s0f3'
for i in $ifc
do
ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 512
tx-usecs 128
ethtool -K $i gro on
ethtool -K $i tso on
done
Server is leaking about 4-6MB per each 10 seconds
MEMLEAK:
5 MB/10sec
6 MB/10sec
4 MB/10sec
4 MB/10sec
And graph where all changes for rx-usecs was done over some time:
https://ibb.co/nrRfbR
Cant eliminate the problem with settings - memleak is bigger or less
visible with rx-usecs set to low values - but then have 100% cpu load - cant
have rx-usecs set to 16
Cant find also other host with same cards or that are using i40e driver
for tests with bisecting
So will just replace to mellanox :)
Also after fresh reboot with i40e
startup settings:
ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
enp3s0f3'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
ethtool -G $i rx 2048 tx 2048
ip link set $i txqueuelen 1000
#ethtool -C $i rx-usecs 256
ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 17
tx-usecs 125
ethtool -L $i combined 6
#ethtool -N $i rx-flow-hash udp4 sdfn
#ethtool -K $i ntuple on
#ethtool -K $i gro off
#ethtool -K $i tso off
done
After issuing:
ethtool -K enp2s0f0 gro on tso on
dmesg shows
[35764.338259] i40e 0000:02:00.0: PF reset failed, -15
and no traffic on the card :)
Also checked now
bigger rx ring
ethtool -G $i rx 2048 tx 2048
Bigger memleag :)
ok need to change cards now to ixgbe .... no reply no help for i40e so
....
maybee someone else with i40e will gather more data i have only this host
soo far - will try to install this cards to other hosts after change but
alll this movement will takes about 2 maybee 3 months - nobody from my team
want to but now cards that supports i40e cause of this bug soo this is hard
now to debug - i need to change also all cards now >10G to mellanox that
have no such bug ... sorry :)
Last tests from my side:)
settings
ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
enp3s0f3'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
ethtool -G $i rx 2048 tx 2048
ip link set $i txqueuelen 1000
ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 17 tx-usecs
125
ethtool -L $i combined 6
ethtool -K $i ntuple on
ethtool -K $i gro on
ethtool -K $i tso on
done
MEMLEAK 1-2MB/10secs
1 MB/10sec
2 MB/10sec
1 MB/10sec
2 MB/10sec
2 MB/10sec
2 MB/10sec
1 MB/10sec
2 MB/10sec
2 MB/10sec
2 MB/10sec
1 MB/10sec
2 MB/10sec
1 MB/10sec
1 MB/10sec
0 MB/10sec
2 MB/10sec
2 MB/10sec
0 MB/10sec
2 MB/10sec
5 MB/10sec
Change rx-usecs 16 tx usecs 16
ifc='enp2s0f0 enp2s0f1 enp2s0f2 enp2s0f3 enp3s0f0 enp3s0f1 enp3s0f2
enp3s0f3'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
ethtool -G $i rx 2048 tx 2048
ip link set $i txqueuelen 1000
ethtool -C $i adaptive-rx off adaptive-tx off rx-usecs 16 tx-usecs
16
ethtool -L $i combined 6
ethtool -K $i ntuple on
ethtool -K $i gro on
ethtool -K $i tso on
done
MEMLEAK: 0-2MB/s with some recycles
0 MB/10sec
0 MB/10sec
0 MB/10sec
0 MB/10sec
0 MB/10sec
0 MB/10sec
1 MB/10sec
0 MB/10sec
2 MB/10sec
0 MB/10sec
2 MB/10sec
-1 MB/10sec
0 MB/10sec
2 MB/10sec
0 MB/10sec
2 MB/10sec
-1 MB/10sec
1 MB/10sec
This data doesn't tell me much of anything and isn't what I asked for.
I don't see how the interrupt throttling rate would be associated with
your memory leak other than possibly rate limiting it by rate limiting
the traffic itself. Is there something that gave you the impression
that interrupt rate was somehow involved?
more interrupts more leak
When we last talked I had asked if you could do a git bisect to find
the memory leak and you said you would look into it. The most useful
way to solve this would be to do a git bisect between your current
kernel and the 4.11 kernel to find the point at which this started. If
we can do that then fixing this becomes much simpler as we just have
to fix the patch that introduced the issue.
Also, I don't know it is you are using to determine that there is a
memory leak. What tool is it you are using to do the tracking? Is
there any specific form of traffic that is causing the leak? If you
can't perform the bisection, any information you could provide that
would allow me to do it would also be useful.
simple script
mem1=`free -m | grep Mem: | awk '{print $3}'`
sleep 10
mem2=`free -m | grep Mem: | awk '{print $3}'`
num=$((mem2 - mem1))
echo $num " MB/10sec"
There is nothing more that gets mem
there is only routed traffic from interface A to B
nothings takes mem
And memleaks only anchge when i change the rx/tx usecs for card
What You need more ?
imagine this is not my only prblem but many - i just want to help i
changed cards to i40e based only cause somebody rises a bug - and i want
to use i40e in feature - dont need them now - but maybee it is good to
help ppl to solwe some problems now if i can - before i will use this
cards ?
I try to use i40e before but there was bug covered by bug - and nobody
from e1000.sf can help me they just reply after year and closing tickets
with info about no activity but they have info in reported bugs ...
soooo what is this ? support center ? for me no .
If i want to help -= after a year response will be something like -
"dont care now" - cause i'v used other hw or sme hacks to repair problem
that should be sloved by intel
Thanks.
- Alex
