[Kernel-packages] [Bug 1388786] Re: TCP stale transfer with erroneous SACK information
This can be fixed by turning off TCP sequence reordering on the Cisco appliance. Please note this also affects your Mac, BSD and Windows machines. You can turn off SACK on your host if you don't care about performance. This feature was enabled by Cisco to protect Windows 95 hosts from TCP sequence prediction attacks (yeah, don't fix the problem, just break the network). However Cisco doesn't translate the SACK ranges it has modified the sequences for so your host gets back the 'wrong' range in the SACK response and simply ignores it because it doesn't match anything it sent. https://supportforums.cisco.com/document/48551/single-tcp-flow- performance-firewall-services-module-fwsm ** Changed in: linux (Ubuntu) Status: In Progress => Invalid -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1388786 Title: TCP stale transfer with erroneous SACK information Status in linux package in Ubuntu: Invalid Bug description: Cisco PIX/FWSM changes TCP sequence numbers but doesn't change numbers in SACK TCP options. When this erroneous information comes to Linux server there is some corruption in TCP stack in some circunstances with CUBIC TCP congestion algorithm and transfer stales. Problem can be reproduced in Ubuntu Server 14.04 when a Cisco FWSM is changing sequence numbers (default configuration) and a big file (30MB, for example) is being transfered. Can be solved deactivating SACK: sysctl -w net.ipv4.tcp_sack=0 We have solved it also with this configuration: sysctl -w net.ipv4.tcp_congestion_control=reno sysctl -w net.ipv4.tcp_frto=1 sysctl -w net.ipv4.tcp_early_retrans=1 We can also fix it by changing firewall configuration. Find attached a wireshark capture where you can see at 16613 frame how client requests segment 853521869 and server (158.42.250.128) resends again a previous segment for 87 seconds until it stops transfer. Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1388786/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1388786] Re: TCP stale transfer with erroneous SACK information
I can reproduce this error too. The environment is a full CISCO network with vmware ESXi hosts and ubuntu 14.04 guests. Also here downloads to about 2MB are going somehow fine, but all which is taking longer (or more MB to transfer) is dropping to a very low bandwidth. Very often without any bits transmitted at all. The issue is, that it looks like if pakets are received in random (not serial order) the SACK seems to be too aggressive and kicks the high speed of the transmission -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1388786 Title: TCP stale transfer with erroneous SACK information Status in linux package in Ubuntu: In Progress Bug description: Cisco PIX/FWSM changes TCP sequence numbers but doesn't change numbers in SACK TCP options. When this erroneous information comes to Linux server there is some corruption in TCP stack in some circunstances with CUBIC TCP congestion algorithm and transfer stales. Problem can be reproduced in Ubuntu Server 14.04 when a Cisco FWSM is changing sequence numbers (default configuration) and a big file (30MB, for example) is being transfered. Can be solved deactivating SACK: sysctl -w net.ipv4.tcp_sack=0 We have solved it also with this configuration: sysctl -w net.ipv4.tcp_congestion_control=reno sysctl -w net.ipv4.tcp_frto=1 sysctl -w net.ipv4.tcp_early_retrans=1 We can also fix it by changing firewall configuration. Find attached a wireshark capture where you can see at 16613 frame how client requests segment 853521869 and server (158.42.250.128) resends again a previous segment for 87 seconds until it stops transfer. Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1388786/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1388786] Re: TCP stale transfer with erroneous SACK information
** Changed in: linux (Ubuntu) Status: Expired = In Progress -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1388786 Title: TCP stale transfer with erroneous SACK information Status in linux package in Ubuntu: In Progress Bug description: Cisco PIX/FWSM changes TCP sequence numbers but doesn't change numbers in SACK TCP options. When this erroneous information comes to Linux server there is some corruption in TCP stack in some circunstances with CUBIC TCP congestion algorithm and transfer stales. Problem can be reproduced in Ubuntu Server 14.04 when a Cisco FWSM is changing sequence numbers (default configuration) and a big file (30MB, for example) is being transfered. Can be solved deactivating SACK: sysctl -w net.ipv4.tcp_sack=0 We have solved it also with this configuration: sysctl -w net.ipv4.tcp_congestion_control=reno sysctl -w net.ipv4.tcp_frto=1 sysctl -w net.ipv4.tcp_early_retrans=1 We can also fix it by changing firewall configuration. Find attached a wireshark capture where you can see at 16613 frame how client requests segment 853521869 and server (158.42.250.128) resends again a previous segment for 87 seconds until it stops transfer. Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1388786/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1388786] Re: TCP stale transfer with erroneous SACK information
[Expired for linux (Ubuntu) because there has been no activity for 60 days.] ** Changed in: linux (Ubuntu) Status: Incomplete = Expired -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1388786 Title: TCP stale transfer with erroneous SACK information Status in linux package in Ubuntu: Expired Bug description: Cisco PIX/FWSM changes TCP sequence numbers but doesn't change numbers in SACK TCP options. When this erroneous information comes to Linux server there is some corruption in TCP stack in some circunstances with CUBIC TCP congestion algorithm and transfer stales. Problem can be reproduced in Ubuntu Server 14.04 when a Cisco FWSM is changing sequence numbers (default configuration) and a big file (30MB, for example) is being transfered. Can be solved deactivating SACK: sysctl -w net.ipv4.tcp_sack=0 We have solved it also with this configuration: sysctl -w net.ipv4.tcp_congestion_control=reno sysctl -w net.ipv4.tcp_frto=1 sysctl -w net.ipv4.tcp_early_retrans=1 We can also fix it by changing firewall configuration. Find attached a wireshark capture where you can see at 16613 frame how client requests segment 853521869 and server (158.42.250.128) resends again a previous segment for 87 seconds until it stops transfer. Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1388786/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1388786] Re: TCP stale transfer with erroneous SACK information
We have tested the kernel 3.19-rc7 and found some improvements. Communication does not stop and the file can be finally downloaded, but it still takes a long time. We have tested a 100 MB file in an Ubuntu server with kernel 3.19-rc7 across the Cisco firewall changing sequence numbers with following results: - With TCP SACK disabled (sysctl -w net.ipv4.tcp_sack=0) 62 seconds - With TCP SACK enabled (default configuration) 462 seconds It seems that, even though the communication is not completely stalled with this kernel version, the problem is not solved yet. ** Tags added: kernel-bug-exists-upstream -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1388786 Title: TCP stale transfer with erroneous SACK information Status in linux package in Ubuntu: Incomplete Bug description: Cisco PIX/FWSM changes TCP sequence numbers but doesn't change numbers in SACK TCP options. When this erroneous information comes to Linux server there is some corruption in TCP stack in some circunstances with CUBIC TCP congestion algorithm and transfer stales. Problem can be reproduced in Ubuntu Server 14.04 when a Cisco FWSM is changing sequence numbers (default configuration) and a big file (30MB, for example) is being transfered. Can be solved deactivating SACK: sysctl -w net.ipv4.tcp_sack=0 We have solved it also with this configuration: sysctl -w net.ipv4.tcp_congestion_control=reno sysctl -w net.ipv4.tcp_frto=1 sysctl -w net.ipv4.tcp_early_retrans=1 We can also fix it by changing firewall configuration. Find attached a wireshark capture where you can see at 16613 frame how client requests segment 853521869 and server (158.42.250.128) resends again a previous segment for 87 seconds until it stops transfer. Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1388786/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1388786] Re: TCP stale transfer with erroneous SACK information
Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.19 kernel[0]. If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'. If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'. If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'. Once testing of the upstream kernel is complete, please mark this bug as Confirmed. Thanks in advance. [0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19-rc7-vivid/ ** Changed in: linux (Ubuntu) Importance: Undecided = Medium ** Changed in: linux (Ubuntu) Status: Confirmed = Incomplete -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1388786 Title: TCP stale transfer with erroneous SACK information Status in linux package in Ubuntu: Incomplete Bug description: Cisco PIX/FWSM changes TCP sequence numbers but doesn't change numbers in SACK TCP options. When this erroneous information comes to Linux server there is some corruption in TCP stack in some circunstances with CUBIC TCP congestion algorithm and transfer stales. Problem can be reproduced in Ubuntu Server 14.04 when a Cisco FWSM is changing sequence numbers (default configuration) and a big file (30MB, for example) is being transfered. Can be solved deactivating SACK: sysctl -w net.ipv4.tcp_sack=0 We have solved it also with this configuration: sysctl -w net.ipv4.tcp_congestion_control=reno sysctl -w net.ipv4.tcp_frto=1 sysctl -w net.ipv4.tcp_early_retrans=1 We can also fix it by changing firewall configuration. Find attached a wireshark capture where you can see at 16613 frame how client requests segment 853521869 and server (158.42.250.128) resends again a previous segment for 87 seconds until it stops transfer. Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1388786/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1388786] Re: TCP stale transfer with erroneous SACK information
I can confirm this bug with the 3.13 kernel shipped with Ubuntu 14.04. The 3.2 kernel shipped with Ubuntu 12.04 did not have the problem, all download were succeeding. The problem started after upgrade to 14.04. The problem also solved after downgrading the kernel to 3.2 by downloading the latest 12.04 (Linux 3.2) one from the PPA. Disabling TCP SACK also resolved the issue. I also tried newer PPA kernels, at least the next one for release with next Ubuntu Update (3.14) does not solve the problem yet. In our case the problems happens mostly with far-away or slower DSL connections, trying to download large files from our webserver. The download starts fine, but suddenly slows down to 0 bytes/sec. It then sometimes recovers, but fails again after a short while. The tcpdump showed, that the client is sending SACKs over and over, but the Ubuntu kernel does not understand them. In our case we also have Cisco hardware inbetween. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1388786 Title: TCP stale transfer with erroneous SACK information Status in linux package in Ubuntu: Confirmed Bug description: Cisco PIX/FWSM changes TCP sequence numbers but doesn't change numbers in SACK TCP options. When this erroneous information comes to Linux server there is some corruption in TCP stack in some circunstances with CUBIC TCP congestion algorithm and transfer stales. Problem can be reproduced in Ubuntu Server 14.04 when a Cisco FWSM is changing sequence numbers (default configuration) and a big file (30MB, for example) is being transfered. Can be solved deactivating SACK: sysctl -w net.ipv4.tcp_sack=0 We have solved it also with this configuration: sysctl -w net.ipv4.tcp_congestion_control=reno sysctl -w net.ipv4.tcp_frto=1 sysctl -w net.ipv4.tcp_early_retrans=1 We can also fix it by changing firewall configuration. Find attached a wireshark capture where you can see at 16613 frame how client requests segment 853521869 and server (158.42.250.128) resends again a previous segment for 87 seconds until it stops transfer. Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1388786/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1388786] Re: TCP stale transfer with erroneous SACK information
This link may also be related, it tells about same problems, but without CISCO hardware: https://askubuntu.com/questions/475700/application- stuck-in-tcp-retransmit/563984 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1388786 Title: TCP stale transfer with erroneous SACK information Status in linux package in Ubuntu: Confirmed Bug description: Cisco PIX/FWSM changes TCP sequence numbers but doesn't change numbers in SACK TCP options. When this erroneous information comes to Linux server there is some corruption in TCP stack in some circunstances with CUBIC TCP congestion algorithm and transfer stales. Problem can be reproduced in Ubuntu Server 14.04 when a Cisco FWSM is changing sequence numbers (default configuration) and a big file (30MB, for example) is being transfered. Can be solved deactivating SACK: sysctl -w net.ipv4.tcp_sack=0 We have solved it also with this configuration: sysctl -w net.ipv4.tcp_congestion_control=reno sysctl -w net.ipv4.tcp_frto=1 sysctl -w net.ipv4.tcp_early_retrans=1 We can also fix it by changing firewall configuration. Find attached a wireshark capture where you can see at 16613 frame how client requests segment 853521869 and server (158.42.250.128) resends again a previous segment for 87 seconds until it stops transfer. Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1388786/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1388786] Re: TCP stale transfer with erroneous SACK information
Ah, in that case you probably wanted to report a bug against the linux package in Ubuntu. I'll move it across. ** Project changed: launchpad = linux (Ubuntu) ** Changed in: linux (Ubuntu) Status: Invalid = New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1388786 Title: TCP stale transfer with erroneous SACK information Status in “linux” package in Ubuntu: Incomplete Bug description: Cisco PIX/FWSM changes TCP sequence numbers but doesn't change numbers in SACK TCP options. When this erroneous information comes to Linux server there is some corruption in TCP stack in some circunstances with CUBIC TCP congestion algorithm and transfer stales. Problem can be reproduced in Ubuntu Server 14.04 when a Cisco FWSM is changing sequence numbers (default configuration) and a big file (30MB, for example) is being transfered. Can be solved deactivating SACK: sysctl -w net.ipv4.tcp_sack=0 We have solved it also with this configuration: sysctl -w net.ipv4.tcp_congestion_control=reno sysctl -w net.ipv4.tcp_frto=1 sysctl -w net.ipv4.tcp_early_retrans=1 We can also fix it by changing firewall configuration. Find attached a wireshark capture where you can see at 16613 frame how client requests segment 853521869 and server (158.42.250.128) resends again a previous segment for 87 seconds until it stops transfer. Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1388786/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1388786] Re: TCP stale transfer with erroneous SACK information
** Changed in: linux (Ubuntu) Status: Incomplete = Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1388786 Title: TCP stale transfer with erroneous SACK information Status in “linux” package in Ubuntu: Confirmed Bug description: Cisco PIX/FWSM changes TCP sequence numbers but doesn't change numbers in SACK TCP options. When this erroneous information comes to Linux server there is some corruption in TCP stack in some circunstances with CUBIC TCP congestion algorithm and transfer stales. Problem can be reproduced in Ubuntu Server 14.04 when a Cisco FWSM is changing sequence numbers (default configuration) and a big file (30MB, for example) is being transfered. Can be solved deactivating SACK: sysctl -w net.ipv4.tcp_sack=0 We have solved it also with this configuration: sysctl -w net.ipv4.tcp_congestion_control=reno sysctl -w net.ipv4.tcp_frto=1 sysctl -w net.ipv4.tcp_early_retrans=1 We can also fix it by changing firewall configuration. Find attached a wireshark capture where you can see at 16613 frame how client requests segment 853521869 and server (158.42.250.128) resends again a previous segment for 87 seconds until it stops transfer. Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1388786/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
