Packet timestamps (was: Re: Network performance degradation from 2.6.11.12 to 2.6.16.20)

2007-03-06 Thread Vladimir B. Savkin
On Fri, Sep 22, 2006 at 09:51:09AM -0700, Rick Jones wrote: That came from named. It opens lots of sockets with SIOCGSTAMP. No idea what it needs that many for. IIRC ISC BIND named opens a socket for each IP it finds on the system. Presumeably in this way it knows implicitly the destination

Re: Packet timestamps (was: Re: Network performance degradation from 2.6.11.12 to 2.6.16.20)

2007-03-06 Thread Eric Dumazet
On Tuesday 06 March 2007 14:25, Vladimir B. Savkin wrote: }, + { + .ctl_name = NET_CORE_ACCURATE_TIMESTAMPS, + .procname = accurate_timestamps, + .data = sysctl_accurate_timestamps, + .maxlen =

Re: Packet timestamps (was: Re: Network performance degradation from 2.6.11.12 to 2.6.16.20)

2007-03-06 Thread Vladimir B. Savkin
On Tue, Mar 06, 2007 at 03:38:44PM +0100, Eric Dumazet wrote: 2) accurate_timestamps is misleading. Should be disable_timestamps Not, if default is 1, as in my patch. ~ :wq With best regards, Vladimir

Re: Packet timestamps (was: Re: Network performance degradation from 2.6.11.12 to 2.6.16.20)

2007-03-06 Thread Eric Dumazet
On Tuesday 06 March 2007 15:43, Vladimir B. Savkin wrote: On Tue, Mar 06, 2007 at 03:38:44PM +0100, Eric Dumazet wrote: 2) accurate_timestamps is misleading. Should be disable_timestamps Not, if default is 1, as in my patch. Yes I saw this. I should write more words next time :) Full

Re: Packet timestamps (was: Re: Network performance degradation from 2.6.11.12 to 2.6.16.20)

2007-03-06 Thread Vladimir B. Savkin
On Tue, Mar 06, 2007 at 04:16:24PM +0100, Eric Dumazet wrote: It would be better to name the tunable disable_timestamps, default 0 of course I agree. If networking maintainers are interested, I surely can prepare a patch. But IMO some way to force TSC usage on x86_64 will be even

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-22 Thread Alexey Kuznetsov
Hello! I can't even find a reference to SIOCGSTAMP in the dhcp-2.0pl5 or dhcp3-3.0.3 sources shipped in Ubuntu. But I will note that tpacket_rcv() expects to always get valid timestamps in the SKB, it does a: It is equally unlikely it uses mmapped packet socket (tpacket_rcv). I even

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-22 Thread Andi Kleen
On Friday 22 September 2006 17:35, Alexey Kuznetsov wrote: Hello! I can't even find a reference to SIOCGSTAMP in the dhcp-2.0pl5 or dhcp3-3.0.3 sources shipped in Ubuntu. But I will note that tpacket_rcv() expects to always get valid timestamps in the SKB, it does a: It is equally

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-22 Thread Rick Jones
That came from named. It opens lots of sockets with SIOCGSTAMP. No idea what it needs that many for. IIRC ISC BIND named opens a socket for each IP it finds on the system. Presumeably in this way it knows implicitly the destination IP without using platform-specific recvfrom/whatever

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-19 Thread Andi Kleen
On Monday 18 September 2006 23:22, David Miller wrote: Ok, ok, but don't we have queueing disciplines that need the timestamp even on ingress? I grepped and I can't find any. The only non SIOCGTSTAMP users of the time stamp seem to be sunrpc and conntrack and I bet both can be converted over

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-19 Thread David Miller
From: Vladimir B. Savkin [EMAIL PROTECTED] Date: Tue, 19 Sep 2006 02:03:31 +0400 On Tue, Sep 19, 2006 at 02:00:38AM +0400, Alexey Kuznetsov wrote: * I do see get_offset_pmtmr() in top lines of profile. That's scary enough. I had it at the very top line. That is just rediculious. You can

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-19 Thread David Miller
From: David Lang [EMAIL PROTECTED] Date: Mon, 18 Sep 2006 14:57:04 -0700 (PDT) yes tcpdump may be wrong in requesting timestamps (in most cases it probably is, but in some cases it's doing exactly what the sysadmin wants it to do), but I don't think that many sysadmins would expect this much

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-19 Thread David Miller
From: Alexey Kuznetsov [EMAIL PROTECTED] Date: Tue, 19 Sep 2006 02:00:38 +0400 * I do not undestand what the hell dhcp needs timestamps for. I can't even find a reference to SIOCGSTAMP in the dhcp-2.0pl5 or dhcp3-3.0.3 sources shipped in Ubuntu. But I will note that tpacket_rcv() expects to

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-19 Thread Stephen Hemminger
The sky2 hardware (and others) can timestamp in hardware, but trying to keep device ticks and system clock in sync looked too nasty to contemplate actually using it. - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-19 Thread Thomas Graf
* David Miller [EMAIL PROTECTED] 2006-09-18 14:22 From: Alexey Kuznetsov [EMAIL PROTECTED] Date: Tue, 19 Sep 2006 01:03:21 +0400 1. It even does not disable possibility to record timestamp inside driver, which Alan was afraid of. The sequence is: if (!skb-tstamp.off_sec)

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-19 Thread Andi Kleen
It seems only natural to me that the real problem is the slow clock source which needs to be resolved regardless of the outcome of this discussion. I believe that updating the stamp at socket enqueue time is the right thing to do but it shouldn't be considered as a solution to the

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Vladimir B. Savkin
On Mon, Sep 18, 2006 at 10:35:38AM +0200, Andi Kleen wrote: I just found out that TSC clocksource is not implemented on x86-64. Kernel version 2.6.18-rc7, is it true? The x86-64 timer subsystems currently doesn't have clocksources at all, but it supports TSC and some other timers. Hm. On

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Andi Kleen
Vladimir B. Savkin [EMAIL PROTECTED] writes: [you seem to send your emails in a strange way that doesn't keep me in cc. Please stop doing that.] On Mon, Sep 18, 2006 at 11:58:21AM +0200, Andi Kleen wrote: The x86-64 timer subsystems currently doesn't have clocksources at all, but it

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread David Miller
From: Andi Kleen [EMAIL PROTECTED] Date: 18 Sep 2006 11:58:21 +0200 For netdev: I'm more and more thinking we should just avoid the problem completely and switch to true end2end timestamps. This means don't time stamp when a packet is received, but only when it is delivered to a socket. The

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Andi Kleen
People who run tcpdump want wire timestamps as close as possible. Yes, things get delayed with the IRQ path, DMA delays, IRQ mitigation and whatnot, but it's an order of magnitude worse if you delay to user read() since that introduces also the delay of the packet copies to userspace which

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Alan Cox
Ar Llu, 2006-09-18 am 16:29 +0200, ysgrifennodd Andi Kleen: The only delay this would add would be the queueing time from the NIC to the softirq. Do you really think that is that bad? If you are trying to do things like network record/playback then you want the minimal delay. There's a reason

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Andi Kleen
On Monday 18 September 2006 17:19, Alan Cox wrote: Ar Llu, 2006-09-18 am 16:29 +0200, ysgrifennodd Andi Kleen: The only delay this would add would be the queueing time from the NIC to the softirq. Do you really think that is that bad? If you are trying to do things like network

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Alexey Kuznetsov
in performance degradation, it just should not be used at all, even such pariah as tcpdump wants to be fast. Actually, I have a question. Why the subject is Network performance degradation from 2.6.11.12 to 2.6.16.20? I do not see beginning of the thread and cannot guess why clock source degraded

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Andi Kleen
to be fast. Actually, I have a question. Why the subject is Network performance degradation from 2.6.11.12 to 2.6.16.20? I do not see beginning of the thread and cannot guess why clock source degraded. :-) It's a long and sad story. Old kernels didn't disable the TSC on those boxes (multi

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Alexey Kuznetsov
Hello! Hmm, not sure how that could happen. Also is it a real problem even if it could? As I said, the problem is _occasionally_ theoretical. This would happen f.e. if packet socket handler was installed after IP handler. Then tcpdump would get packet after it is processed

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Andi Kleen
On Monday 18 September 2006 18:28, Alexey Kuznetsov wrote: Hello! Hmm, not sure how that could happen. Also is it a real problem even if it could? As I said, the problem is _occasionally_ theoretical. This would happen f.e. if packet socket handler was installed after IP handler.

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Alexey Kuznetsov
Hello! But that never happens right? Right. Well, not right. It happens. Simply because you get packet with newer timestamp after previous handler saw this packet and did some actions. I just do not see any bad consequences. And do you have some other prefered way to solve this? Even if

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Vladimir B. Savkin
On Mon, Sep 18, 2006 at 01:27:57PM +0200, Andi Kleen wrote: The codebase for timing (and lots of other things) is quite different between 32bit and 64bit. You're really surprised it doesn't work if you do such things? It works, and after your remark above, I'm surprised. Dunno about slow TSC

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Vladimir B. Savkin
On Mon, Sep 18, 2006 at 06:50:22PM +0200, Andi Kleen wrote: I suppose in the worst case a sysctl like Vladimir asked for could be added, but it would seem somewhat lame. Please think about it this way: suppose you haave a heavily loaded router and some network problem is to be diagnosed. You

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread David Miller
From: Alexey Kuznetsov [EMAIL PROTECTED] Date: Tue, 19 Sep 2006 01:03:21 +0400 1. It even does not disable possibility to record timestamp inside driver, which Alan was afraid of. The sequence is: if (!skb-tstamp.off_sec) net_timestamp(skb); 2. Maybe, netif_rx()

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Alexey Kuznetsov
Hello! Please think about it this way: suppose you haave a heavily loaded router and some network problem is to be diagnosed. You run tcpdump and suddenly router becomes overloaded (by switching to timestamp-it-all mode I am sorry. I cannot think that way. :-) Instead of attempts to scare,

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Vladimir B. Savkin
On Tue, Sep 19, 2006 at 02:00:38AM +0400, Alexey Kuznetsov wrote: Hello! Please think about it this way: suppose you haave a heavily loaded router and some network problem is to be diagnosed. You run tcpdump and suddenly router becomes overloaded (by switching to timestamp-it-all mode

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread David Lang
On Tue, 19 Sep 2006, Alexey Kuznetsov wrote: Hello! Please think about it this way: suppose you haave a heavily loaded router and some network problem is to be diagnosed. You run tcpdump and suddenly router becomes overloaded (by switching to timestamp-it-all mode I am sorry. I cannot think

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Andi Kleen
On Monday 18 September 2006 23:03, Alexey Kuznetsov wrote: And do you have some other prefered way to solve this? Even if the timer was fast it would be still good to avoid it in the fast path when DHCPD is running. No. The way, which you suggested, seems to be the best. Ok. I also

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-07-10 Thread Jesper Dangaard Brouer
On Tue, 4 Jul 2006, Andi Kleen wrote: On Tuesday 04 July 2006 13:41, Jesper Dangaard Brouer wrote: Actually the change happens between kernel version 2.6.15 and 2.6.16. The timestamp optimizations are older. Don't remember the exact release, but earlier 2.6. What I'm saying is that,

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-07-04 Thread Jesper Dangaard Brouer
On Mon, 26 Jun 2006, Andi Kleen wrote: I encountered the same problem on a dual core opteron equipped with a broadcom NIC (tg3) under 2.4. It could receive 1 Mpps when using TSC as the clock source, but the time jumped back and forth, so I changed it to 'notsc', then the performance dropped

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-07-04 Thread Andi Kleen
On Tuesday 04 July 2006 13:41, Jesper Dangaard Brouer wrote: On Mon, 26 Jun 2006, Andi Kleen wrote: I encountered the same problem on a dual core opteron equipped with a broadcom NIC (tg3) under 2.4. It could receive 1 Mpps when using TSC as the clock source, but the time jumped back

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-25 Thread Harry Edmon
I understand the saying beggars can't be choosers, but I have heard nothing on this issue since June 19th. Does anyone have any ideas on what is going on? Is there more information I can collect that would help diagnose this problem? And again, thanks for any and all help! -- Dr. Harry

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-25 Thread Willy Tarreau
Hi Andi, On Mon, Jun 19, 2006 at 05:24:31PM +0200, Andi Kleen wrote: If you use pmtmr try to reboot with kernel option clock=tsc. That's dangerous advice - when the system choses not to use TSC it often has a reason. On my Opteron AMD system i normally can route 400 kpps, but with

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-25 Thread Bill Fink
On Sun, 25 Jun 2006, Harry Edmon wrote: I understand the saying beggars can't be choosers, but I have heard nothing on this issue since June 19th. Does anyone have any ideas on what is going on? Is there more information I can collect that would help diagnose this problem? And

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-25 Thread Andi Kleen
I encountered the same problem on a dual core opteron equipped with a broadcom NIC (tg3) under 2.4. It could receive 1 Mpps when using TSC as the clock source, but the time jumped back and forth, so I changed it to 'notsc', then the performance dropped dramatically to around the same value

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-19 Thread Harry Edmon
Stephen Hemminger wrote: Does this fix it? # sysctl -w net.ipv4.tcp_abc=0 That did not help. I have 1 minute outputs from tcpdump under both 2.6.11.12 and 2.6.16.20. You will see a large size difference between the files. Since the 2.6.11.12 one is 2 MBytes, I thought I would post

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-19 Thread Jesper Dangaard Brouer
Harry Edmon [EMAIL PROTECTED] wrote: I have a system with a strange network performance degradation from 2.6.11.12 to most recent kernels including 2.6.16.20 and 2.6.17-rc6. The system is has Dual single core Xeons with hyperthreading on. cut Hi Harry Can you check which high-res

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-19 Thread Harry Edmon
Jesper Dangaard Brouer wrote: Harry Edmon [EMAIL PROTECTED] wrote: I have a system with a strange network performance degradation from 2.6.11.12 to most recent kernels including 2.6.16.20 and 2.6.17-rc6. The system is has Dual single core Xeons with hyperthreading on. cut Hi Harry Can

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-19 Thread Chris Friesen
Andi Kleen wrote: Incoming packets are only time stamped when someone asks for the timestamps. Doesn't that add scheduling latency to the timestamps? Or is is a flag that gets set to trigger timestamping at packet arrival? Chris - To unsubscribe from this list: send the line unsubscribe

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-19 Thread Jesper Dangaard Brouer
On Mon, 19 Jun 2006, Andi Kleen wrote: If you use pmtmr try to reboot with kernel option clock=tsc. That's dangerous advice - when the system choses not to use TSC it often has a reason. Sorry, it was not a general advice, just something to try out. It really solved my network

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-19 Thread Andi Kleen
On Monday 19 June 2006 19:34, Chris Friesen wrote: Andi Kleen wrote: Incoming packets are only time stamped when someone asks for the timestamps. Doesn't that add scheduling latency to the timestamps? Or is is a flag that gets set to trigger timestamping at packet arrival? It's a flag

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-19 Thread Herbert Xu
Harry Edmon [EMAIL PROTECTED] wrote: That did not help. I have 1 minute outputs from tcpdump under both 2.6.11.12 and 2.6.16.20. You will see a large size difference between the files. Since the 2.6.11.12 one is 2 MBytes, I thought I would post them via the web instead of via

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-18 Thread Harry Edmon
Stephen Hemminger wrote: Does this fix it? # sysctl -w net.ipv4.tcp_abc=0 Thanks for the suggestion. I will give it a try later tonight. Also Andrew - sorry for the incorrect placement of my follow-up comments. I do appreciate everyone's help in figuring this out. -- Dr. Harry

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-17 Thread Andrew Morton
On Fri, 16 Jun 2006 09:01:23 -0700 Harry Edmon [EMAIL PROTECTED] wrote: I have a system with a strange network performance degradation from 2.6.11.12 to most recent kernels including 2.6.16.20 and 2.6.17-rc6. The system is has Dual single core Xeons with hyperthreading on. The

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-17 Thread Harry Edmon
I assume you are talking about using TCP_NODELAY as a socket option within the LDM software. I could give that a try. There is a lot of traffic on this node, on the order of 2000 packets in and out per second, so the tcpdump output will grow pretty fast. How long a tcpdump would be useful,

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-17 Thread Andrew Morton
On Sat, 17 Jun 2006 16:23:34 -0700 Harry Edmon [EMAIL PROTECTED] wrote: Andrew Morton wrote: On Fri, 16 Jun 2006 09:01:23 -0700 Harry Edmon [EMAIL PROTECTED] wrote: I have a system with a strange network performance degradation from 2.6.11.12 to most recent kernels including

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-17 Thread Stephen Hemminger
Andrew Morton wrote: On Sat, 17 Jun 2006 16:23:34 -0700 Harry Edmon [EMAIL PROTECTED] wrote: Andrew Morton wrote: On Fri, 16 Jun 2006 09:01:23 -0700 Harry Edmon [EMAIL PROTECTED] wrote: I have a system with a strange network performance degradation from 2.6.11.12 to most