Re: [PATCH] apparmor: Fix network performance issue in aa_label_sk_perm

2018-09-07 Thread Tony Jones
On 09/07/2018 09:37 AM, John Johansen wrote: > hey Tony, > > thanks for the patch, I am curious did you're investigation look > into what parts of DEFINE_AUDIT_SK are causing the issue? Hi JJ. Attached are the perf annotations for DEFINE_AUDIT_SK (percentages are relative to the fn). Our

Re: [PATCH] apparmor: Fix network performance issue in aa_label_sk_perm

2018-09-07 Thread Tony Jones
On 09/07/2018 09:37 AM, John Johansen wrote: > hey Tony, > > thanks for the patch, I am curious did you're investigation look > into what parts of DEFINE_AUDIT_SK are causing the issue? Hi JJ. Attached are the perf annotations for DEFINE_AUDIT_SK (percentages are relative to the fn). Our

Re: [PATCH] apparmor: Fix network performance issue in aa_label_sk_perm

2018-09-07 Thread John Johansen
On 09/06/2018 09:33 PM, Tony Jones wrote: > The netperf benchmark shows a 5.73% reduction in throughput for > small (64 byte) transfers by unconfined tasks. > > DEFINE_AUDIT_SK() in aa_label_sk_perm() should not be performed > unconditionally, rather only when the label is confined. > >

Re: [PATCH] apparmor: Fix network performance issue in aa_label_sk_perm

2018-09-07 Thread John Johansen
On 09/06/2018 09:33 PM, Tony Jones wrote: > The netperf benchmark shows a 5.73% reduction in throughput for > small (64 byte) transfers by unconfined tasks. > > DEFINE_AUDIT_SK() in aa_label_sk_perm() should not be performed > unconditionally, rather only when the label is confined. > >

[PATCH] apparmor: Fix network performance issue in aa_label_sk_perm

2018-09-06 Thread Tony Jones
The netperf benchmark shows a 5.73% reduction in throughput for small (64 byte) transfers by unconfined tasks. DEFINE_AUDIT_SK() in aa_label_sk_perm() should not be performed unconditionally, rather only when the label is confined. netperf-tcp 56974a6fc^

[PATCH] apparmor: Fix network performance issue in aa_label_sk_perm

2018-09-06 Thread Tony Jones
The netperf benchmark shows a 5.73% reduction in throughput for small (64 byte) transfers by unconfined tasks. DEFINE_AUDIT_SK() in aa_label_sk_perm() should not be performed unconditionally, rather only when the label is confined. netperf-tcp 56974a6fc^

Re: network performance get regression from 2.6 to 3.10 by each version

2014-05-05 Thread Rick Jones
On 05/02/2014 12:40 PM, V JobNickname wrote: I have an ARM platform which works with older 2.6.28 Linux Kernel and the embedded NIC driver I profile the TCP Tx using netperf 2.6 by command "./netperf -H {serverip} -l 300". Is your ARM platform a multi-core one? If so, you may need/want to

Re: network performance get regression from 2.6 to 3.10 by each version

2014-05-05 Thread Rick Jones
On 05/02/2014 12:40 PM, V JobNickname wrote: I have an ARM platform which works with older 2.6.28 Linux Kernel and the embedded NIC driver I profile the TCP Tx using netperf 2.6 by command ./netperf -H {serverip} -l 300. Is your ARM platform a multi-core one? If so, you may need/want to look

network performance get regression from 2.6 to 3.10 by each version

2014-05-02 Thread V JobNickname
I have an ARM platform which works with older 2.6.28 Linux Kernel and the embedded NIC driver I profile the TCP Tx using netperf 2.6 by command "./netperf -H {serverip} -l 300". In 2.6.28 the TCP tx can reach 190 Mbps. Recently I am porting the platform to long-term Kernel version 2.6.32.61,

network performance get regression from 2.6 to 3.10 by each version

2014-05-02 Thread V JobNickname
I have an ARM platform which works with older 2.6.28 Linux Kernel and the embedded NIC driver I profile the TCP Tx using netperf 2.6 by command ./netperf -H {serverip} -l 300. In 2.6.28 the TCP tx can reach 190 Mbps. Recently I am porting the platform to long-term Kernel version 2.6.32.61,

Re: Poor network performance x86_64.. also with 3.13

2014-02-09 Thread Borislav Petkov
0 was fine even with these settings and 3.12 wasn't. Here's the original report: "I recently upgraded the Kernel from version 3.10 to latest stable 3.12.8, did the usual "make oldconfig" (resulting config attached). But now I noticed some _really_ low network performance." Link

Re: Poor network performance x86_64.. also with 3.13

2014-02-09 Thread Eric Dumazet
On Sun, 2014-02-09 at 16:31 +0100, Borislav Petkov wrote: > On Sun, Feb 09, 2014 at 04:05:11PM +0100, Daniel Exner wrote: > > > cat /etc/sysctl.d/net.conf > > > net.ipv4.tcp_window_scaling = 1 > > > net.core.rmem_max = 16777216 > > > net.ipv4.tcp_rmem = 4096 87380 16777 > > > net.ipv4.tcp_wmem =

Re: Poor network performance x86_64.. also with 3.13

2014-02-09 Thread Borislav Petkov
On Sun, Feb 09, 2014 at 04:05:11PM +0100, Daniel Exner wrote: > > cat /etc/sysctl.d/net.conf > > net.ipv4.tcp_window_scaling = 1 > > net.core.rmem_max = 16777216 > > net.ipv4.tcp_rmem = 4096 87380 16777 > > net.ipv4.tcp_wmem = 4096 1638 > > After removing those values I finally had sane

Re: Poor network performance x86_64.. also with 3.13

2014-02-09 Thread Daniel Exner
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi all, Am Mon, 20 Jan 2014 23:37:52 +0100 schrieb Borislav Petkov : > On Mon, Jan 20, 2014 at 11:27:25PM +0100, Daniel Exner wrote: > > I just did the same procedure with Kernel Version 3.13: same poor > > rates. > > > > I think I will try to

Re: Poor network performance x86_64.. also with 3.13

2014-02-09 Thread Daniel Exner
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi all, Am Mon, 20 Jan 2014 23:37:52 +0100 schrieb Borislav Petkov b...@alien8.de: On Mon, Jan 20, 2014 at 11:27:25PM +0100, Daniel Exner wrote: I just did the same procedure with Kernel Version 3.13: same poor rates. I think I will try

Re: Poor network performance x86_64.. also with 3.13

2014-02-09 Thread Borislav Petkov
On Sun, Feb 09, 2014 at 04:05:11PM +0100, Daniel Exner wrote: cat /etc/sysctl.d/net.conf net.ipv4.tcp_window_scaling = 1 net.core.rmem_max = 16777216 net.ipv4.tcp_rmem = 4096 87380 16777 net.ipv4.tcp_wmem = 4096 1638 After removing those values I finally had sane iperf values.

Re: Poor network performance x86_64.. also with 3.13

2014-02-09 Thread Eric Dumazet
On Sun, 2014-02-09 at 16:31 +0100, Borislav Petkov wrote: On Sun, Feb 09, 2014 at 04:05:11PM +0100, Daniel Exner wrote: cat /etc/sysctl.d/net.conf net.ipv4.tcp_window_scaling = 1 net.core.rmem_max = 16777216 net.ipv4.tcp_rmem = 4096 87380 16777 net.ipv4.tcp_wmem = 4096 1638

Re: Poor network performance x86_64.. also with 3.13

2014-02-09 Thread Borislav Petkov
with these settings and 3.12 wasn't. Here's the original report: I recently upgraded the Kernel from version 3.10 to latest stable 3.12.8, did the usual make oldconfig (resulting config attached). But now I noticed some _really_ low network performance. Link: http://lkml.kernel.org/r/52dad66f.7080

Re: Poor network performance x86_64.. also with 3.13

2014-01-20 Thread Branimir Maksimovic
On 01/20/2014 11:37 PM, Borislav Petkov wrote: On Mon, Jan 20, 2014 at 11:27:25PM +0100, Daniel Exner wrote: I just did the same procedure with Kernel Version 3.13: same poor rates. I think I will try to see of 3.12.6 was still ok and bisect from there. Or try something more coarse-grained

Re: Poor network performance x86_64.. also with 3.13

2014-01-20 Thread Borislav Petkov
On Mon, Jan 20, 2014 at 11:27:25PM +0100, Daniel Exner wrote: > I just did the same procedure with Kernel Version 3.13: same poor rates. > > I think I will try to see of 3.12.6 was still ok and bisect from there. Or try something more coarse-grained like 3.11 first, then 3.12 and then the -rcs

Poor network performance x86_64.. also with 3.13

2014-01-20 Thread Daniel Exner
ne.linux.kernel.] > >> On Sat, 18 Jan 2014 20:30:55 +0100, Daniel Exner wrote: > >>> I recently upgraded the Kernel from version 3.10 to latest >>> stable 3.12.8, did the usual "make oldconfig" (resulting >>> config attached). >>>

Poor network performance x86_64.. also with 3.13

2014-01-20 Thread Daniel Exner
20:30:55 +0100, Daniel Exner wrote: I recently upgraded the Kernel from version 3.10 to latest stable 3.12.8, did the usual make oldconfig (resulting config attached). But now I noticed some _really_ low network performance. Try: sysctl net.ipv4.tcp_limit_output_bytes=262144 Tried

Re: Poor network performance x86_64.. also with 3.13

2014-01-20 Thread Borislav Petkov
On Mon, Jan 20, 2014 at 11:27:25PM +0100, Daniel Exner wrote: I just did the same procedure with Kernel Version 3.13: same poor rates. I think I will try to see of 3.12.6 was still ok and bisect from there. Or try something more coarse-grained like 3.11 first, then 3.12 and then the -rcs in

Re: Poor network performance x86_64.. also with 3.13

2014-01-20 Thread Branimir Maksimovic
On 01/20/2014 11:37 PM, Borislav Petkov wrote: On Mon, Jan 20, 2014 at 11:27:25PM +0100, Daniel Exner wrote: I just did the same procedure with Kernel Version 3.13: same poor rates. I think I will try to see of 3.12.6 was still ok and bisect from there. Or try something more coarse-grained

Re: 3.12.8 poor network performance x86_64

2014-01-18 Thread Daniel Exner
ote: > >> I recently upgraded the Kernel from version 3.10 to latest >> stable 3.12.8, did the usual "make oldconfig" (resulting config >> attached). >> >> But now I noticed some _really_ low network performance. > > Try: sysctl net.ipv4.tcp_limit_o

Re: 3.12.8 poor network performance x86_64

2014-01-18 Thread Holger Hoffstätte
On Sat, 18 Jan 2014 20:30:55 +0100, Daniel Exner wrote: > I recently upgraded the Kernel from version 3.10 to latest stable > 3.12.8, did the usual "make oldconfig" (resulting config attached). > > But now I noticed some _really_ low network performance.

Re: 3.12.8 poor network performance x86_64

2014-01-18 Thread Holger Hoffstätte
On Sat, 18 Jan 2014 20:30:55 +0100, Daniel Exner wrote: I recently upgraded the Kernel from version 3.10 to latest stable 3.12.8, did the usual make oldconfig (resulting config attached). But now I noticed some _really_ low network performance. Try: sysctl net.ipv4.tcp_limit_output_bytes

Re: 3.12.8 poor network performance x86_64

2014-01-18 Thread Daniel Exner
upgraded the Kernel from version 3.10 to latest stable 3.12.8, did the usual make oldconfig (resulting config attached). But now I noticed some _really_ low network performance. Try: sysctl net.ipv4.tcp_limit_output_bytes=262144 Tried that. Even 10 times the value. Same effect

Re: Major network performance regression in 3.7

2013-01-06 Thread John Stoffel
> "Willy" == Willy Tarreau writes: Willy> On Sun, Jan 06, 2013 at 11:00:15AM -0800, Eric Dumazet wrote: >> On Sun, 2013-01-06 at 10:51 -0800, Eric Dumazet wrote: >> > > >> > > (sd->len is usually 4096, which is expected, but sd->total_len value is >> > > huge in your case, so we always set

Re: Major network performance regression in 3.7

2013-01-06 Thread John Stoffel
> "Willy" == Willy Tarreau writes: Willy> On Sun, Jan 06, 2013 at 04:49:35PM -0500, John Stoffel wrote: >> > "Willy" == Willy Tarreau writes: >> Willy> On Sun, Jan 06, 2013 at 11:00:15AM -0800, Eric Dumazet wrote: >> >> On Sun, 2013-01-06 at 10:51 -0800, Eric Dumazet wrote: >> >> > >

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 04:49:35PM -0500, John Stoffel wrote: > > "Willy" == Willy Tarreau writes: > > Willy> On Sun, Jan 06, 2013 at 11:00:15AM -0800, Eric Dumazet wrote: > >> On Sun, 2013-01-06 at 10:51 -0800, Eric Dumazet wrote: > >> > > > >> > > (sd->len is usually 4096, which is

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 11:39:31AM -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 20:34 +0100, Willy Tarreau wrote: > > > OK it works like a charm here now ! I can't break it anymore, so it > > looks like you finally got it ! > > > > I noticed that the data rate was higher when the

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 20:34 +0100, Willy Tarreau wrote: > OK it works like a charm here now ! I can't break it anymore, so it > looks like you finally got it ! > > I noticed that the data rate was higher when the loopback's MTU > is exactly a multiple of 4096 (making the 64k choice optimal) >

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 11:00:15AM -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 10:51 -0800, Eric Dumazet wrote: > > > > > > (sd->len is usually 4096, which is expected, but sd->total_len value is > > > huge in your case, so we always set the flag in fs/splice.c) > > > > I am testing : > >

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 10:51 -0800, Eric Dumazet wrote: > > > > (sd->len is usually 4096, which is expected, but sd->total_len value is > > huge in your case, so we always set the flag in fs/splice.c) > > I am testing : > >if (sd->len < sd->total_len && pipe->nrbufs > 1) >

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
> > (sd->len is usually 4096, which is expected, but sd->total_len value is > huge in your case, so we always set the flag in fs/splice.c) I am testing : if (sd->len < sd->total_len && pipe->nrbufs > 1) more |= MSG_SENDPAGE_NOTLAST; -- To unsubscribe from this list:

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 10:39 -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 18:35 +0100, Willy Tarreau wrote: > > > Unfortunately it does not work any better, which means to me > > that we don't leave via this code path. I tried other tricks > > which failed too. I need to understand this part

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 18:35 +0100, Willy Tarreau wrote: > Unfortunately it does not work any better, which means to me > that we don't leave via this code path. I tried other tricks > which failed too. I need to understand this part better before > randomly fiddling with it. > OK, now I have

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 09:10:55AM -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 17:44 +0100, Willy Tarreau wrote: > > On Sun, Jan 06, 2013 at 08:39:53AM -0800, Eric Dumazet wrote: > > > Hmm, I'll have to check if this really can be reverted without hurting > > > vmsplice() again. > > > >

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 17:44 +0100, Willy Tarreau wrote: > On Sun, Jan 06, 2013 at 08:39:53AM -0800, Eric Dumazet wrote: > > Hmm, I'll have to check if this really can be reverted without hurting > > vmsplice() again. > > Looking at the code I've been wondering whether we shouldn't transform > the

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 16:51 +0100, Willy Tarreau wrote: > Hi Eric, > > Oh sorry, I didn't really want to pollute the list with links and configs, > especially during the initial report with various combined issues :-( > > The client is my old "inject" tool, available here : > >

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 08:39:53AM -0800, Eric Dumazet wrote: > Hmm, I'll have to check if this really can be reverted without hurting > vmsplice() again. Looking at the code I've been wondering whether we shouldn't transform the condition to perform the push if we can't push more segments, but I

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
Hi Eric, On Sun, Jan 06, 2013 at 06:59:02AM -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 10:24 +0100, Willy Tarreau wrote: > > > It does not change anything to the tests above unfortunately. It did not > > even stabilize the unstable runs. > > > > I'll check if I can spot the original

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 10:24 +0100, Willy Tarreau wrote: > It does not change anything to the tests above unfortunately. It did not > even stabilize the unstable runs. > > I'll check if I can spot the original commit which caused the regression > for MTUs that are not n*4096+52. Since you don't

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 11:25:25AM +0100, Willy Tarreau wrote: > OK good news here, the performance drop on the myri was caused by a > problem between the keyboard and the chair. After the reboot series, > I forgot to reload the firmware so the driver used the less efficient > firmware from the

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 12:46:58PM +0100, Romain Francoise wrote: > Willy Tarreau writes: > > > That makes me think that I should try 3.8-rc2 since LRO was removed > > there :-/ > > Better yet, find a way to automate these tests so they can run continually > against net-next and find problems

Re: Major network performance regression in 3.7

2013-01-06 Thread Romain Francoise
Willy Tarreau writes: > That makes me think that I should try 3.8-rc2 since LRO was removed > there :-/ Better yet, find a way to automate these tests so they can run continually against net-next and find problems early... -- To unsubscribe from this list: send the line "unsubscribe

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 10:24:35AM +0100, Willy Tarreau wrote: > But before that I'll try to find the recent one causing the myri10ge to > slow down, it should take less time to bisect. OK good news here, the performance drop on the myri was caused by a problem between the keyboard and the chair.

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sat, Jan 05, 2013 at 11:35:24PM -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 03:52 +0100, Willy Tarreau wrote: > > > OK so I observed no change with this patch, either on the loopback > > data rate at >16kB MTU, or on the myri. I'm keeping it at hand for > > experimentation anyway. > >

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sat, Jan 05, 2013 at 11:35:24PM -0800, Eric Dumazet wrote: On Sun, 2013-01-06 at 03:52 +0100, Willy Tarreau wrote: OK so I observed no change with this patch, either on the loopback data rate at 16kB MTU, or on the myri. I'm keeping it at hand for experimentation anyway. Yeah,

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 10:24:35AM +0100, Willy Tarreau wrote: But before that I'll try to find the recent one causing the myri10ge to slow down, it should take less time to bisect. OK good news here, the performance drop on the myri was caused by a problem between the keyboard and the chair.

Re: Major network performance regression in 3.7

2013-01-06 Thread Romain Francoise
Willy Tarreau w...@1wt.eu writes: That makes me think that I should try 3.8-rc2 since LRO was removed there :-/ Better yet, find a way to automate these tests so they can run continually against net-next and find problems early... -- To unsubscribe from this list: send the line unsubscribe

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 12:46:58PM +0100, Romain Francoise wrote: Willy Tarreau w...@1wt.eu writes: That makes me think that I should try 3.8-rc2 since LRO was removed there :-/ Better yet, find a way to automate these tests so they can run continually against net-next and find problems

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 11:25:25AM +0100, Willy Tarreau wrote: OK good news here, the performance drop on the myri was caused by a problem between the keyboard and the chair. After the reboot series, I forgot to reload the firmware so the driver used the less efficient firmware from the NIC

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 10:24 +0100, Willy Tarreau wrote: It does not change anything to the tests above unfortunately. It did not even stabilize the unstable runs. I'll check if I can spot the original commit which caused the regression for MTUs that are not n*4096+52. Since you don't post

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
Hi Eric, On Sun, Jan 06, 2013 at 06:59:02AM -0800, Eric Dumazet wrote: On Sun, 2013-01-06 at 10:24 +0100, Willy Tarreau wrote: It does not change anything to the tests above unfortunately. It did not even stabilize the unstable runs. I'll check if I can spot the original commit which

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 08:39:53AM -0800, Eric Dumazet wrote: Hmm, I'll have to check if this really can be reverted without hurting vmsplice() again. Looking at the code I've been wondering whether we shouldn't transform the condition to perform the push if we can't push more segments, but I

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 16:51 +0100, Willy Tarreau wrote: Hi Eric, Oh sorry, I didn't really want to pollute the list with links and configs, especially during the initial report with various combined issues :-( The client is my old inject tool, available here :

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 17:44 +0100, Willy Tarreau wrote: On Sun, Jan 06, 2013 at 08:39:53AM -0800, Eric Dumazet wrote: Hmm, I'll have to check if this really can be reverted without hurting vmsplice() again. Looking at the code I've been wondering whether we shouldn't transform the

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 09:10:55AM -0800, Eric Dumazet wrote: On Sun, 2013-01-06 at 17:44 +0100, Willy Tarreau wrote: On Sun, Jan 06, 2013 at 08:39:53AM -0800, Eric Dumazet wrote: Hmm, I'll have to check if this really can be reverted without hurting vmsplice() again. Looking at the

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 18:35 +0100, Willy Tarreau wrote: Unfortunately it does not work any better, which means to me that we don't leave via this code path. I tried other tricks which failed too. I need to understand this part better before randomly fiddling with it. OK, now I have your

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 10:39 -0800, Eric Dumazet wrote: On Sun, 2013-01-06 at 18:35 +0100, Willy Tarreau wrote: Unfortunately it does not work any better, which means to me that we don't leave via this code path. I tried other tricks which failed too. I need to understand this part better

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
(sd-len is usually 4096, which is expected, but sd-total_len value is huge in your case, so we always set the flag in fs/splice.c) I am testing : if (sd-len sd-total_len pipe-nrbufs 1) more |= MSG_SENDPAGE_NOTLAST; -- To unsubscribe from this list: send the line

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 10:51 -0800, Eric Dumazet wrote: (sd-len is usually 4096, which is expected, but sd-total_len value is huge in your case, so we always set the flag in fs/splice.c) I am testing : if (sd-len sd-total_len pipe-nrbufs 1) more |=

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 11:00:15AM -0800, Eric Dumazet wrote: On Sun, 2013-01-06 at 10:51 -0800, Eric Dumazet wrote: (sd-len is usually 4096, which is expected, but sd-total_len value is huge in your case, so we always set the flag in fs/splice.c) I am testing : if

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 20:34 +0100, Willy Tarreau wrote: OK it works like a charm here now ! I can't break it anymore, so it looks like you finally got it ! I noticed that the data rate was higher when the loopback's MTU is exactly a multiple of 4096 (making the 64k choice optimal) while I

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 11:39:31AM -0800, Eric Dumazet wrote: On Sun, 2013-01-06 at 20:34 +0100, Willy Tarreau wrote: OK it works like a charm here now ! I can't break it anymore, so it looks like you finally got it ! I noticed that the data rate was higher when the loopback's MTU is

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 04:49:35PM -0500, John Stoffel wrote: Willy == Willy Tarreau w...@1wt.eu writes: Willy On Sun, Jan 06, 2013 at 11:00:15AM -0800, Eric Dumazet wrote: On Sun, 2013-01-06 at 10:51 -0800, Eric Dumazet wrote: (sd-len is usually 4096, which is expected, but

Re: Major network performance regression in 3.7

2013-01-06 Thread John Stoffel
Willy == Willy Tarreau w...@1wt.eu writes: Willy On Sun, Jan 06, 2013 at 04:49:35PM -0500, John Stoffel wrote: Willy == Willy Tarreau w...@1wt.eu writes: Willy On Sun, Jan 06, 2013 at 11:00:15AM -0800, Eric Dumazet wrote: On Sun, 2013-01-06 at 10:51 -0800, Eric Dumazet wrote:

Re: Major network performance regression in 3.7

2013-01-06 Thread John Stoffel
Willy == Willy Tarreau w...@1wt.eu writes: Willy On Sun, Jan 06, 2013 at 11:00:15AM -0800, Eric Dumazet wrote: On Sun, 2013-01-06 at 10:51 -0800, Eric Dumazet wrote: (sd-len is usually 4096, which is expected, but sd-total_len value is huge in your case, so we always set the flag in

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sun, 2013-01-06 at 03:52 +0100, Willy Tarreau wrote: > OK so I observed no change with this patch, either on the loopback > data rate at >16kB MTU, or on the myri. I'm keeping it at hand for > experimentation anyway. > Yeah, there was no bug. I rewrote it for net-next as a cleanup/optim

Re: Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
On Sat, Jan 05, 2013 at 06:16:31PM -0800, Eric Dumazet wrote: > On Sat, 2013-01-05 at 17:51 -0800, Eric Dumazet wrote: > > On Sat, 2013-01-05 at 17:40 -0800, Eric Dumazet wrote: > > > On Sun, 2013-01-06 at 02:30 +0100, Willy Tarreau wrote: > > > > > > > Ah interesting because these were some of

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sun, 2013-01-06 at 03:32 +0100, Willy Tarreau wrote: > It's 0cf833ae (net: loopback: set default mtu to 64K). And I could > reproduce it with 3.6 by setting loopback's MTU to 65536 by hand. > The trick is that once the MTU has been set to this large a value, > even when I set it back to 16kB

Re: Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
On Sat, Jan 05, 2013 at 06:22:13PM -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 03:18 +0100, Willy Tarreau wrote: > > On Sat, Jan 05, 2013 at 06:16:31PM -0800, Eric Dumazet wrote: > > > On Sat, 2013-01-05 at 17:51 -0800, Eric Dumazet wrote: > > > > On Sat, 2013-01-05 at 17:40 -0800, Eric

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sun, 2013-01-06 at 03:18 +0100, Willy Tarreau wrote: > On Sat, Jan 05, 2013 at 06:16:31PM -0800, Eric Dumazet wrote: > > On Sat, 2013-01-05 at 17:51 -0800, Eric Dumazet wrote: > > > On Sat, 2013-01-05 at 17:40 -0800, Eric Dumazet wrote: > > > > On Sun, 2013-01-06 at 02:30 +0100, Willy Tarreau

Re: Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
On Sat, Jan 05, 2013 at 06:16:31PM -0800, Eric Dumazet wrote: > On Sat, 2013-01-05 at 17:51 -0800, Eric Dumazet wrote: > > On Sat, 2013-01-05 at 17:40 -0800, Eric Dumazet wrote: > > > On Sun, 2013-01-06 at 02:30 +0100, Willy Tarreau wrote: > > > > > > > Ah interesting because these were some of

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sat, 2013-01-05 at 17:51 -0800, Eric Dumazet wrote: > On Sat, 2013-01-05 at 17:40 -0800, Eric Dumazet wrote: > > On Sun, 2013-01-06 at 02:30 +0100, Willy Tarreau wrote: > > > > > Ah interesting because these were some of the mm patches that I had > > > tried to revert. > > > > Hmm, or we

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sat, 2013-01-05 at 17:40 -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 02:30 +0100, Willy Tarreau wrote: > > > Ah interesting because these were some of the mm patches that I had > > tried to revert. > > Hmm, or we should fix __skb_splice_bits() > > I'll send a patch. > Could you try

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sun, 2013-01-06 at 02:30 +0100, Willy Tarreau wrote: > Ah interesting because these were some of the mm patches that I had > tried to revert. Hmm, or we should fix __skb_splice_bits() I'll send a patch. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body

Re: Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
On Sat, Jan 05, 2013 at 05:21:16PM -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 01:50 +0100, Willy Tarreau wrote: > > > Yes, I've removed all zero counters in this short view for easier > > reading (complete version appended at the end of this email). This > > was after around 140 GB were

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sun, 2013-01-06 at 01:50 +0100, Willy Tarreau wrote: > Yes, I've removed all zero counters in this short view for easier > reading (complete version appended at the end of this email). This > was after around 140 GB were transferred : OK I only wanted to make sure skb were not linearized in

Re: Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
On Sat, Jan 05, 2013 at 04:02:03PM -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 00:29 +0100, Willy Tarreau wrote: > > > > 2) Another possibility would be that Myri card/driver doesnt like very > > > well high order pages. > > > > It looks like it has not changed much since 3.6 :-/ I really

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sun, 2013-01-06 at 00:29 +0100, Willy Tarreau wrote: > > 2) Another possibility would be that Myri card/driver doesnt like very > > well high order pages. > > It looks like it has not changed much since 3.6 :-/ I really suspect > something is wrong with memory allocation. I have tried

Re: Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
Hi Eric, On Sat, Jan 05, 2013 at 03:18:46PM -0800, Eric Dumazet wrote: > Hi Willy, another good finding during the week end ! ;) Yes, I wanted to experiment with TFO and stopped on this :-) > 1) This looks like interrupts are spreaded on multiple cpus, and this > give Out Of Order problems with

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sat, 2013-01-05 at 22:49 +0100, Willy Tarreau wrote: > Hi, > > I'm observing multiple apparently unrelated network performance > issues in 3.7, to the point that I'm doubting it comes from the > network stack. > > My setup involves 3 machines connected point-to-point

Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
Hi, I'm observing multiple apparently unrelated network performance issues in 3.7, to the point that I'm doubting it comes from the network stack. My setup involves 3 machines connected point-to-point with myri 10GE NICs (the middle machine has 2 NICs). The middle machine normally runs haproxy

Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
Hi, I'm observing multiple apparently unrelated network performance issues in 3.7, to the point that I'm doubting it comes from the network stack. My setup involves 3 machines connected point-to-point with myri 10GE NICs (the middle machine has 2 NICs). The middle machine normally runs haproxy

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sat, 2013-01-05 at 22:49 +0100, Willy Tarreau wrote: Hi, I'm observing multiple apparently unrelated network performance issues in 3.7, to the point that I'm doubting it comes from the network stack. My setup involves 3 machines connected point-to-point with myri 10GE NICs (the middle

Re: Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
Hi Eric, On Sat, Jan 05, 2013 at 03:18:46PM -0800, Eric Dumazet wrote: Hi Willy, another good finding during the week end ! ;) Yes, I wanted to experiment with TFO and stopped on this :-) 1) This looks like interrupts are spreaded on multiple cpus, and this give Out Of Order problems with

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sun, 2013-01-06 at 00:29 +0100, Willy Tarreau wrote: 2) Another possibility would be that Myri card/driver doesnt like very well high order pages. It looks like it has not changed much since 3.6 :-/ I really suspect something is wrong with memory allocation. I have tried reverting many

Re: Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
On Sat, Jan 05, 2013 at 04:02:03PM -0800, Eric Dumazet wrote: On Sun, 2013-01-06 at 00:29 +0100, Willy Tarreau wrote: 2) Another possibility would be that Myri card/driver doesnt like very well high order pages. It looks like it has not changed much since 3.6 :-/ I really suspect

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sun, 2013-01-06 at 01:50 +0100, Willy Tarreau wrote: Yes, I've removed all zero counters in this short view for easier reading (complete version appended at the end of this email). This was after around 140 GB were transferred : OK I only wanted to make sure skb were not linearized in

Re: Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
On Sat, Jan 05, 2013 at 05:21:16PM -0800, Eric Dumazet wrote: On Sun, 2013-01-06 at 01:50 +0100, Willy Tarreau wrote: Yes, I've removed all zero counters in this short view for easier reading (complete version appended at the end of this email). This was after around 140 GB were

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sun, 2013-01-06 at 02:30 +0100, Willy Tarreau wrote: Ah interesting because these were some of the mm patches that I had tried to revert. Hmm, or we should fix __skb_splice_bits() I'll send a patch. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sat, 2013-01-05 at 17:40 -0800, Eric Dumazet wrote: On Sun, 2013-01-06 at 02:30 +0100, Willy Tarreau wrote: Ah interesting because these were some of the mm patches that I had tried to revert. Hmm, or we should fix __skb_splice_bits() I'll send a patch. Could you try the

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sat, 2013-01-05 at 17:51 -0800, Eric Dumazet wrote: On Sat, 2013-01-05 at 17:40 -0800, Eric Dumazet wrote: On Sun, 2013-01-06 at 02:30 +0100, Willy Tarreau wrote: Ah interesting because these were some of the mm patches that I had tried to revert. Hmm, or we should fix

Re: Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
On Sat, Jan 05, 2013 at 06:16:31PM -0800, Eric Dumazet wrote: On Sat, 2013-01-05 at 17:51 -0800, Eric Dumazet wrote: On Sat, 2013-01-05 at 17:40 -0800, Eric Dumazet wrote: On Sun, 2013-01-06 at 02:30 +0100, Willy Tarreau wrote: Ah interesting because these were some of the mm patches

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sun, 2013-01-06 at 03:18 +0100, Willy Tarreau wrote: On Sat, Jan 05, 2013 at 06:16:31PM -0800, Eric Dumazet wrote: On Sat, 2013-01-05 at 17:51 -0800, Eric Dumazet wrote: On Sat, 2013-01-05 at 17:40 -0800, Eric Dumazet wrote: On Sun, 2013-01-06 at 02:30 +0100, Willy Tarreau wrote:

Re: Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
On Sat, Jan 05, 2013 at 06:22:13PM -0800, Eric Dumazet wrote: On Sun, 2013-01-06 at 03:18 +0100, Willy Tarreau wrote: On Sat, Jan 05, 2013 at 06:16:31PM -0800, Eric Dumazet wrote: On Sat, 2013-01-05 at 17:51 -0800, Eric Dumazet wrote: On Sat, 2013-01-05 at 17:40 -0800, Eric Dumazet

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sun, 2013-01-06 at 03:32 +0100, Willy Tarreau wrote: It's 0cf833ae (net: loopback: set default mtu to 64K). And I could reproduce it with 3.6 by setting loopback's MTU to 65536 by hand. The trick is that once the MTU has been set to this large a value, even when I set it back to 16kB the

  1   2   >