Re: TCP send/ack lockstep problem
Hi, found it. This is the tcpdump, for your edification. Note that many packets have "push" set for no good reason, which generally indicates dropped packets if we're doing a bulk transfer. But all packets that are visible in the dump reached their destination..! The culprit turns out to be the traffic shaper, which had a subtly-broken configuration and thus shaped that particular traffic with a scythe instead of a razor. A way to attach tcpdump so that those dropped packets get reported (with an appropriate indication) would probably be helpful. Anyway, this is not a packet that was dropped on the wire. So why set the Push flag? TCP *knows* it couldn' send the thing. 09:16:40.652962 IP (tos 0x0, ttl 53, id 16206, offset 0, flags [DF], proto: TCP (6), length: 60) 62.193.238.120.58818 > 192.109.102.42.3306: S, cksum 0x4ef3 (correct), 692009621:692009621(0) win 5840 09:16:40.653030 IP (tos 0x34, ttl 64, id 0, offset 0, flags [DF], proto: TCP (6), length: 60) 192.109.102.42.3306 > 62.193.238.120.58818: S, cksum 0xf277 (correct), 3232139371:3232139371(0) ack 692009622 win 5696 09:16:40.705214 IP (tos 0x0, ttl 53, id 16207, offset 0, flags [DF], proto: TCP (6), length: 52) 62.193.238.120.58818 > 192.109.102.42.3306: ., cksum 0x302f (correct), 1:1(0) ack 1 win 1460 09:16:40.716501 IP (tos 0x38, ttl 64, id 32802, offset 0, flags [DF], proto: TCP (6), length: 127) 192.109.102.42.3306 > 62.193.238.120.58818: P 1:76(75) ack 1 win 1424 09:16:40.754853 IP (tos 0x0, ttl 53, id 16208, offset 0, flags [DF], proto: TCP (6), length: 52) 62.193.238.120.58818 > 192.109.102.42.3306: ., cksum 0x2fac (correct), 1:1(0) ack 76 win 1460 09:16:40.758899 IP (tos 0x0, ttl 53, id 16209, offset 0, flags [DF], proto: TCP (6), length: 116) 62.193.238.120.58818 > 192.109.102.42.3306: P 1:65(64) ack 76 win 1460 09:16:40.759877 IP (tos 0x34, ttl 64, id 32803, offset 0, flags [DF], proto: TCP (6), length: 52) 192.109.102.42.3306 > 62.193.238.120.58818: ., cksum 0x2f88 (correct), 76:76(0) ack 65 win 1424 09:16:40.768414 IP (tos 0x34, ttl 64, id 32804, offset 0, flags [DF], proto: TCP (6), length: 63) 192.109.102.42.3306 > 62.193.238.120.58818: P, cksum 0x2870 (correct), 76:87(11) ack 65 win 1424 09:16:40.805883 IP (tos 0x0, ttl 53, id 16210, offset 0, flags [DF], proto: TCP (6), length: 73) 62.193.238.120.58818 > 192.109.102.42.3306: P, cksum 0xd2f7 (correct), 65:86(21) ack 87 win 1460 09:16:40.808077 IP (tos 0x38, ttl 64, id 32805, offset 0, flags [DF], proto: TCP (6), length: 140) 192.109.102.42.3306 > 62.193.238.120.58818: P 87:175(88) ack 86 win 1424 09:16:40.847493 IP (tos 0x0, ttl 53, id 16211, offset 0, flags [DF], proto: TCP (6), length: 83) 62.193.238.120.58818 > 192.109.102.42.3306: P 86:117(31) ack 175 win 1460 09:16:40.855500 IP (tos 0x38, ttl 64, id 32806, offset 0, flags [DF], proto: TCP (6), length: 124) 192.109.102.42.3306 > 62.193.238.120.58818: P 175:247(72) ack 117 win 1424 09:16:40.855701 IP (tos 0x34, ttl 64, id 32807, offset 0, flags [DF], proto: TCP (6), length: 52) 192.109.102.42.3306 > 62.193.238.120.58818: F, cksum 0x2e46 (correct), 247:247(0) ack 117 win 1424 09:16:40.896780 IP (tos 0x0, ttl 53, id 16212, offset 0, flags [DF], proto: TCP (6), length: 57) 62.193.238.120.58818 > 192.109.102.42.3306: P, cksum 0x2be4 (correct), 117:122(5) ack 247 win 1460 09:16:40.896811 IP (tos 0x34, ttl 64, id 0, offset 0, flags [DF], proto: TCP (6), length: 40) 192.109.102.42.3306 > 62.193.238.120.58818: R, cksum 0x1f59 (correct), 3232139618:3232139618(0) win 0 09:16:40.897032 IP (tos 0x0, ttl 53, id 16213, offset 0, flags [DF], proto: TCP (6), length: 52) 62.193.238.120.58818 > 192.109.102.42.3306: F, cksum 0x2deb (correct), 122:122(0) ack 247 win 1460 09:16:40.897058 IP (tos 0x34, ttl 64, id 0, offset 0, flags [DF], proto: TCP (6), length: 40) 192.109.102.42.3306 > 62.193.238.120.58818: R, cksum 0x1f59 (correct), 3232139618:3232139618(0) win 0 09:16:40.898497 IP (tos 0x0, ttl 53, id 16214, offset 0, flags [DF], proto: TCP (6), length: 52) 62.193.238.120.58818 > 192.109.102.42.3306: ., cksum 0x2de8 (correct), 123:123(0) ack 248 win 1460 09:16:40.898517 IP (tos 0x34, ttl 64, id 0, offset 0, flags [DF], proto: TCP (6), length: 40) 192.109.102.42.3306 > 62.193.238.120.58818: R, cksum 0x1f58 (correct), 3232139619:3232139619(0) win 0 09:16:48.515549 IP (tos 0x0, ttl 53, id 35709, offset 0, flags [DF], proto: TCP (6), length: 60) 62.193.238.120.58819 > 192.109.102.42.3306: S, cksum 0xbb47 (correct), 685289454:685289454(0) win 5840 09:16:48.515601 IP (tos 0x34, ttl 64, id 0, offset 0, flags [DF], proto: TCP (6), length: 60) 192.109.102.42.3306 > 62.193.238.120.58819: S, cksum 0x824a (correct), 3248316644:3248316644(0) ack 685289455 win 5696 09:16:48.551765 IP (tos 0x0, ttl 53, id 35710, offset 0, flags [DF], proto: TCP (6), length: 52) 62.193.238.120.58819 > 192.109.102.42.3306: ., cksum 0xc011 (correct), 1:1(0) ack 1 win 1460 09:16:48.555452 IP (to
Re: TCP send/ack lockstep problem
Matthias Urlichs wrote: Hi, Herbert Xu: Matthias Urlichs <[EMAIL PROTECTED]> wrote: Any ideas? Is the sending program doing something stupid? If so, what? Probably. Since you're on the sender's side (192.109.x.x I presume) why don't you do a strace on the sending process and tell us? The stuff I see in there doesn't look stupid to me. accept(...) = 216 setsockopt(216, SOL_IP, IP_TOS, [8], 4) = 0 setsockopt(216, SOL_TCP, TCP_NODELAY, [1], 4) = 0 setsockopt(216, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0 followed by a bunch of "interactive" read() and write() calls, and then a rather long sequence of 16384-byte writes (all of which end up being partial because the send buffer size is 12k). so if one of these is the cause for the observed lockstep behavior when the send buffer is *full*, I'd consider that to be a bug. Interestingly, the *sender* advertizes window that's (a bit more than) one MTU: 07:47:03.204939 IP (tos 0x38, ttl 64, id 41217, offset 0, flags [DF], proto: TCP (6), length: 1468) 192.109.102.42.3306 > 62.193.238.120.53138: . 167089:168505 (1416) ack 0 win 1424 but that's the other direction -- is something in the kernel confused about which window variable to use ?? You could try turning off window scaling: sysctl -w net.ipv4.tcp_window_scaling=0 If that fixes it, you have a corrupting middlebox in the path or bad implementation on the other end. Also try turning off tcp_abc (on the sender). sysctl -w net.ipv4.tcp_abc=0 -- VGER BF report: H 0.050318 - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: TCP send/ack lockstep problem
Matthias Urlichs <[EMAIL PROTECTED]> wrote:
>
> The stuff I see in there doesn't look stupid to me.
>
> accept(...) = 216
> setsockopt(216, SOL_IP, IP_TOS, [8], 4) = 0
> setsockopt(216, SOL_TCP, TCP_NODELAY, [1], 4) = 0
> setsockopt(216, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
> followed by a bunch of "interactive" read() and write() calls, and then
> a rather long sequence of 16384-byte writes (all of which end up being
> partial because the send buffer size is 12k).
Please send the actual strace with the options -ttT. If you can also
send the tcpdumps that occur at the same time then we can match them up
and figure out what's going on.
So far the only thing I can see from your original tcpdump is that every
other ack is followed by a noticeable delay before a packet is sent.
This is a typical symptom when the appliation has nothing in the buffer.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[EMAIL PROTECTED]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
--
VGER BF report: U 0.499395
-
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: TCP send/ack lockstep problem
Hi,
Herbert Xu:
> Matthias Urlichs <[EMAIL PROTECTED]> wrote:
> >
> > Any ideas? Is the sending program doing something stupid? If so, what?
>
> Probably. Since you're on the sender's side (192.109.x.x I presume) why
> don't you do a strace on the sending process and tell us?
>
The stuff I see in there doesn't look stupid to me.
accept(...) = 216
setsockopt(216, SOL_IP, IP_TOS, [8], 4) = 0
setsockopt(216, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(216, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
followed by a bunch of "interactive" read() and write() calls, and then
a rather long sequence of 16384-byte writes (all of which end up being
partial because the send buffer size is 12k).
so if one of these is the cause for the observed lockstep behavior when
the send buffer is *full*, I'd consider that to be a bug.
Interestingly, the *sender* advertizes window that's (a bit more than) one MTU:
07:47:03.204939 IP (tos 0x38, ttl 64, id 41217, offset 0, flags [DF], proto:
TCP (6), length: 1468) 192.109.102.42.3306 > 62.193.238.120.53138: .
167089:168505 (1416) ack 0 win 1424
but that's the other direction -- is something in the kernel confused about
which window variable to use ??
--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | [EMAIL PROTECTED]
--
VGER BF report: U 0.499405
-
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: TCP send/ack lockstep problem
Matthias Urlichs <[EMAIL PROTECTED]> wrote:
>
> Any ideas? Is the sending program doing something stupid? If so, what?
Probably. Since you're on the sender's side (192.109.x.x I presume) why
don't you do a strace on the sending process and tell us?
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[EMAIL PROTECTED]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
--
VGER BF report: U 0.52
-
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
TCP send/ack lockstep problem
Hello, I have a problem with a TCP connection here which just doesn't want to have multiple packets on the wire. I have verified that the sender has enough buffered data (netstat -t shows 8..10k send buffer). There's no packet loss. The tcpdump (attached) shows that the receiver increases its window correctly -- the sender just seems to ignore that. Any ideas? Is the sending program doing something stupid? If so, what? tcpdump (on the sender's interface; Ubuntu kernel 2.6.15-23): 11:00:02.101523 IP (tos 0x34, ttl 53, id 19121, offset 0, flags [DF], proto: TCP (6), length: 60) 62.193.238.120.33126 > 192.109.102.42.3306: S, cksum 0x130b (correct), 3222543538:3222543538(0) win 5840 11:00:02.101566 IP (tos 0x34, ttl 64, id 0, offset 0, flags [DF], proto: TCP (6), length: 60) 192.109.102.42.3306 > 62.193.238.120.33126: S, cksum 0xcff6 (correct), 1482874191:1482874191(0) ack 3222543539 win 5696 11:00:02.149295 IP (tos 0x34, ttl 53, id 50966, offset 0, flags [DF], proto: TCP (6), length: 52) 62.193.238.120.33115 > 192.109.102.42.3306: ., cksum 0x2703 (correct), 86:86(0) ack 99 win 1460 11:00:02.153222 IP (tos 0x34, ttl 53, id 19122, offset 0, flags [DF], proto: TCP (6), length: 52) 62.193.238.120.33126 > 192.109.102.42.3306: ., cksum 0x0daf (correct), 1:1(0) ack 1 win 1460 11:00:02.153504 IP (tos 0x38, ttl 64, id 56076, offset 0, flags [DF], proto: TCP (6), length: 127) 192.109.102.42.3306 > 62.193.238.120.33126: P 1:76(75) ack 1 win 1424 11:00:02.166282 IP (tos 0x38, ttl 64, id 27087, offset 0, flags [DF], proto: TCP (6), length: 1468) 192.109.102.42.3306 > 62.193.238.120.33821: P 1015273:1016689(1416) ack 0 win 1424 11:00:02.190953 IP (tos 0x34, ttl 53, id 19123, offset 0, flags [DF], proto: TCP (6), length: 52) 62.193.238.120.33126 > 192.109.102.42.3306: ., cksum 0x0d39 (correct), 1:1(0) ack 76 win 1460 11:00:02.194676 IP (tos 0x0, ttl 53, id 19124, offset 0, flags [DF], proto: TCP (6), length: 116) 62.193.238.120.33126 > 192.109.102.42.3306: P 1:65(64) ack 76 win 1460 11:00:02.194727 IP (tos 0x34, ttl 64, id 56077, offset 0, flags [DF], proto: TCP (6), length: 52) 192.109.102.42.3306 > 62.193.238.120.33126: ., cksum 0x0d15 (correct), 76:76(0) ack 65 win 1424 11:00:02.194835 IP (tos 0x34, ttl 64, id 56078, offset 0, flags [DF], proto: TCP (6), length: 63) 192.109.102.42.3306 > 62.193.238.120.33126: P, cksum 0x05fe (correct), 76:87(11) ack 65 win 1424 11:00:02.223927 IP (tos 0x34, ttl 53, id 64017, offset 0, flags [DF], proto: TCP (6), length: 52) 62.193.238.120.33821 > 192.109.102.42.3306: ., cksum 0xd02b (correct), 0:0(0) ack 1016689 win 16022 11:00:02.223961 IP (tos 0x38, ttl 64, id 27088, offset 0, flags [DF], proto: TCP (6), length: 1468) 192.109.102.42.3306 > 62.193.238.120.33821: P 1016689:1018105(1416) ack 0 win 1424 11:00:02.236030 IP (tos 0x0, ttl 53, id 19125, offset 0, flags [DF], proto: TCP (6), length: 73) 62.193.238.120.33126 > 192.109.102.42.3306: P, cksum 0xcf80 (correct), 65:86(21) ack 87 win 1460 11:00:02.240355 IP (tos 0x38, ttl 64, id 56079, offset 0, flags [DF], proto: TCP (6), length: 1468) 192.109.102.42.3306 > 62.193.238.120.33126: . 87:1503(1416) ack 86 win 1424 11:00:02.281126 IP (tos 0x34, ttl 53, id 64018, offset 0, flags [DF], proto: TCP (6), length: 52) 62.193.238.120.33821 > 192.109.102.42.3306: ., cksum 0xca65 (correct), 0:0(0) ack 1018105 win 16022 11:00:02.375243 IP (tos 0x34, ttl 53, id 19126, offset 0, flags [DF], proto: TCP (6), length: 52) 62.193.238.120.33126 > 192.109.102.42.3306: ., cksum 0x03bb (correct), 86:86(0) ack 1503 win 2184 11:00:02.385427 IP (tos 0x38, ttl 64, id 56080, offset 0, flags [DF], proto: TCP (6), length: 1468) 192.109.102.42.3306 > 62.193.238.120.33126: . 1503:2919(1416) ack 86 win 1424 11:00:02.444747 IP (tos 0x34, ttl 53, id 19127, offset 0, flags [DF], proto: TCP (6), length: 52) 62.193.238.120.33126 > 192.109.102.42.3306: ., cksum 0xfb19 (correct), 86:86(0) ack 2919 win 2908 11:00:02.444787 IP (tos 0x38, ttl 64, id 56082, offset 0, flags [DF], proto: TCP (6), length: 1468) 192.109.102.42.3306 > 62.193.238.120.33126: P 2919:4335(1416) ack 86 win 1424 11:00:02.530953 IP (tos 0x34, ttl 53, id 19128, offset 0, flags [DF], proto: TCP (6), length: 52) 62.193.238.120.33126 > 192.109.102.42.3306: ., cksum 0xf253 (correct), 86:86(0) ack 4335 win 3632 11:00:02.626367 IP (tos 0x38, ttl 64, id 56083, offset 0, flags [DF], proto: TCP (6), length: 1468) 192.109.102.42.3306 > 62.193.238.120.33126: . 4335:5751(1416) ack 86 win 1424 11:00:02.685415 IP (tos 0x34, ttl 53, id 19129, offset 0, flags [DF], proto: TCP (6), length: 52) 62.193.238.120.33126 > 192.109.102.42.3306: ., cksum 0xe953 (correct), 86:86(0) ack 5751 win 4356 11:00:02.685459 IP (tos 0x38, ttl 64, id 56084, offset 0, flags [DF], proto: TCP (6), length: 1468) 192.109.102.42.3306 > 62.193.238.120.33126: . 5751:7167(1416) ack 86 win 1424 11:00:02.741868 IP (tos 0x34, ttl 53, id 19130, offset 0, flags [DF]
