Re: Linux ECN Handling

Hi Neal, I've attached a pdf of a slide that I made which shows some data from the kernel log at the receiver as well as the timestamp, SeqNo, and length of the corresponding segments from the tcpdump trace at the receiver interface. Hopefully this helps clarify why I think tcp_transmit_skb() is c

Re: Linux ECN Handling

On Tue, Jan 2, 2018 at 6:57 PM, Steve Ibanez wrote: > Hi Neal, > > Sorry, my last email was incorrect. It turns out the default tcp > congestion control alg that was being used on my client machines was > cubic instead of dctcp. That is why tp->processing_cwr field was never > set in the tcp_rcv_e

Re: Linux ECN Handling

Hi Neal, Sorry, my last email was incorrect. It turns out the default tcp congestion control alg that was being used on my client machines was cubic instead of dctcp. That is why tp->processing_cwr field was never set in the tcp_rcv_established function. I've changed the default back to dctcp on a

Re: Linux ECN Handling

On Tue, Jan 2, 2018 at 2:43 AM, Steve Ibanez wrote: > Hi Neal, > > Apologies for the delay, and happy new year! > > To answer your question, data is only transferred in one direction > (from the client to the server). The SeqNo in the pkts from the server > to the client is not incremented. So I d

Re: Linux ECN Handling

Hi Neal, Apologies for the delay, and happy new year! To answer your question, data is only transferred in one direction (from the client to the server). The SeqNo in the pkts from the server to the client is not incremented. So I don't think that a data pkt is attempted to be sent in the tcp_dat

Re: Linux ECN Handling

On Wed, Dec 20, 2017 at 2:20 PM, Steve Ibanez wrote: > > Hi Neal, > > I added in some more printk statements and it does indeed look like > all of these calls you listed are being invoked successfully. I guess > this isn't too surprising given what the inet_csk_schedule_ack() and > inet_csk_ack_sc

Re: Linux ECN Handling

Hi Neal, I added in some more printk statements and it does indeed look like all of these calls you listed are being invoked successfully. I guess this isn't too surprising given what the inet_csk_schedule_ack() and inet_csk_ack_scheduled() functions are doing: static inline void inet_csk_schedul

Re: Linux ECN Handling

On Tue, Dec 19, 2017 at 5:00 PM, Steve Ibanez wrote: > Hi Neal, > > I managed to track down the code path that the unACKed CWR packet is > taking. The tcp_rcv_established() function calls tcp_ack_snd_check() > at the end of step5 and then the return statement indicated below is > invoked, which pr

Re: Linux ECN Handling

Hi Neal, I managed to track down the code path that the unACKed CWR packet is taking. The tcp_rcv_established() function calls tcp_ack_snd_check() at the end of step5 and then the return statement indicated below is invoked, which prevents the __tcp_ack_snd_check() function from running. static i

Re: Linux ECN Handling

On Tue, Dec 19, 2017 at 12:16 AM, Steve Ibanez wrote: > > Hi Neal, > > I started looking into this receiver ACKing issue today. Strangely, > when I tried adding printk statements at the top of the > tcp_v4_do_rcv(), tcp_rcv_established(), __tcp_ack_snd_check() and > tcp_send_delayed_ack() function

Re: Linux ECN Handling

Hi Neal, I started looking into this receiver ACKing issue today. Strangely, when I tried adding printk statements at the top of the tcp_v4_do_rcv(), tcp_rcv_established(), __tcp_ack_snd_check() and tcp_send_delayed_ack() functions they were never executed on the machine running the iperf3 server

Re: Linux ECN Handling

On Tue, Dec 5, 2017 at 2:36 PM, Steve Ibanez wrote: > Hi Neal, > > I've included a link to small trace of 13 packets which is different > from the screenshot I attached in my last email, but shows the same > sequence of events. It's a bit hard to read the tcptrace due to the > 300ms timeout, so I

Re: Linux ECN Handling

Hi Neal, I've included a link to small trace of 13 packets which is different from the screenshot I attached in my last email, but shows the same sequence of events. It's a bit hard to read the tcptrace due to the 300ms timeout, so I figured this was the best approach. slice.pcap: https://drive.g

Re: Linux ECN Handling

On Tue, Dec 5, 2017 at 12:22 AM, Steve Ibanez wrote: > Hi Neal, > > Happy to help out :) And thanks for the tip! > > I was able to track down where the missing bytes that you pointed out > are being lost. It turns out the destination host seems to be > misbehaving. I performed a packet capture at

Re: Linux ECN Handling

On Mon, Nov 27, 2017 at 1:49 PM, Steve Ibanez wrote: > > Hi Neal, > > I tried out your new suggested patches and indeed it looks like it is > working. The duration of the freezes looks like it has reduced from an > RTO to 10ms (tcp probe reports SRTT measurements of about 200us just > before the f

Re: Linux ECN Handling

Hi Neal, I tried out your new suggested patches and indeed it looks like it is working. The duration of the freezes looks like it has reduced from an RTO to 10ms (tcp probe reports SRTT measurements of about 200us just before the freeze). So the PTO value seems to be correctly set to max(2*SRTT, 1

Re: Linux ECN Handling

On Tue, Nov 21, 2017 at 10:02 PM, Steve Ibanez wrote: > Hi Neal, > > I just tried out your fix for enabling TLPs in the CWR state (while > leaving tcp_tso_should_defer() unchanged), but I'm still seeing the > host enter long timeouts. Feel free to let me know if there is > something else you'd lik

Re: Linux ECN Handling

Hi Neal, I just tried out your fix for enabling TLPs in the CWR state (while leaving tcp_tso_should_defer() unchanged), but I'm still seeing the host enter long timeouts. Feel free to let me know if there is something else you'd like me to try. Best, -Steve On Tue, Nov 21, 2017 at 8:52 AM, Eric

Re: Linux ECN Handling

On Tue, Nov 21, 2017 at 8:20 AM, Neal Cardwell wrote: > On Tue, Nov 21, 2017 at 10:51 AM, Yuchung Cheng wrote: >> On Tue, Nov 21, 2017 at 7:01 AM, Neal Cardwell wrote: >>> >>> The original motivation for only allowing TLP in the CA_Open state was >>> to be conservative and avoid having the TLP i

Re: Linux ECN Handling

On Tue, Nov 21, 2017 at 10:51 AM, Yuchung Cheng wrote: > On Tue, Nov 21, 2017 at 7:01 AM, Neal Cardwell wrote: >> >> The original motivation for only allowing TLP in the CA_Open state was >> to be conservative and avoid having the TLP impose extra load on the >> bottleneck when it may be congeste

Re: Linux ECN Handling

On Tue, Nov 21, 2017 at 7:01 AM, Neal Cardwell wrote: > > On Tue, Nov 21, 2017 at 12:58 AM, Steve Ibanez wrote: > > Hi Neal, > > > > I tried your suggestion to disable tcp_tso_should_defer() and it does > > indeed look like it is preventing the host from entering timeouts. > > I'll have to do a b

Re: Linux ECN Handling

On Tue, Nov 21, 2017 at 12:58 AM, Steve Ibanez wrote: > Hi Neal, > > I tried your suggestion to disable tcp_tso_should_defer() and it does > indeed look like it is preventing the host from entering timeouts. > I'll have to do a bit more digging to try and find where the packets > are being dropped

Re: Linux ECN Handling

Hi Neal, I tried your suggestion to disable tcp_tso_should_defer() and it does indeed look like it is preventing the host from entering timeouts. I'll have to do a bit more digging to try and find where the packets are being dropped. I've verified that the bottleneck link queue is capacity is at a

Re: Linux ECN Handling

On Mon, Nov 20, 2017 at 7:01 AM, Neal Cardwell wrote: > Going back to one of your Oct 19 trace snapshots (attached), AFAICT at the > time of the timeout there is actually almost 64KBytes (352553398 + 1448 - > 352489686 = 65160) of unacknowledged data. So there really does seem to be a > significa

Re: Linux ECN Handling

On Mon, Nov 20, 2017 at 2:31 AM, Steve Ibanez wrote: > Hi Folks, > > I wanted to check back in on this for another update and to solicit > some more suggestions. I did a bit more digging to try an isolate the > problem. Going back to one of your Oct 19 trace snapshots (attached), AFAICT at the ti

Re: Linux ECN Handling

Hi Folks, I wanted to check back in on this for another update and to solicit some more suggestions. I did a bit more digging to try an isolate the problem. As I explained earlier, the log generated by tcp_probe indicates that the snd_cwnd is set to 1 just before the end host receives an ECN mark

Re: Linux ECN Handling

Hi Daniel, Apologies for the delay. I tried out Neal's suggestion to printk the cwnd and ss-thresh in the tcp_init_cwnd_reduction(), tcp_cwnd_reduction(), and tcp_end_cwnd_reduction() functions in tcp_input.c. From what I can tell, none of these functions are setting the cwnd to 0. Here is the ke

Re: Linux ECN Handling

On 10/24/2017 03:11 AM, Neal Cardwell wrote: On Mon, Oct 23, 2017 at 6:15 PM, Steve Ibanez wrote: Hi All, I upgraded the kernel on all of our machines to Linux 4.13.8-041308-lowlatency. However, I'm still observing the same behavior where the source enters a timeout when the CWND=1MSS and it r

Re: Linux ECN Handling

On Mon, Oct 23, 2017 at 6:15 PM, Steve Ibanez wrote: > Hi All, > > I upgraded the kernel on all of our machines to Linux > 4.13.8-041308-lowlatency. However, I'm still observing the same > behavior where the source enters a timeout when the CWND=1MSS and it > receives ECN marks. > > Here are the m

Re: Linux ECN Handling

Hi All, I upgraded the kernel on all of our machines to Linux 4.13.8-041308-lowlatency. However, I'm still observing the same behavior where the source enters a timeout when the CWND=1MSS and it receives ECN marks. Here are the measured flow rates:

Re: Linux ECN Handling

[ full-quoting due to Cc fixups, adding netdev ] Steve Ibanez wrote: > Hi Florian, Neal, and Daniel, > > I hope this email finds you well. My name is Stephen Ibanez and I'm a PhD > Student at Stanford currently working on a project with Mohammad Alizadeh, > Nick McKeown, and Lavanya Jose. We hav