another reassemble tcp problem - details for PF developers
There has been 2 recent threads mentioning problems with reassemble tcp: pf: reassemble tcp problems with emails through pf For info here is another. We solved the problem by removing this scrub. We went from 4.3 with scrub in all to the latest 4.6 stable with match in all scrub (reassemble tcp) and bizarrely found we sometimes (~50%) couldnt FTP to several sites. It would hang just before sending any data. This would happen about 50% of the time and work fine the rest. It may be an irrelevant coincidence but each FTP site that this happened with was Microsoft FTP. It never happened (ie FTP always worked fine) with other server types. The OpenBSD firewall is running ftp-proxy. The internal client is a linux desktop. Removing the reassemble tcp scrub made the problem go away and a successful connection/download every time. We took tcpdumps on both the internal and external interfaces of the OpenBSD firewall/ftp-proxy. Looking at our capture on firewall internal interface, the payload of the first TCP packet after the handshake seems to be ignored. I see: Frames 26, 28, 29: TCP handshake 33: first data from server 34: client then immediately sends ACK to server with seq=1 35: last of the data from server 36: server attempting to close connection 37, 38: Client sends two ACKs in quick succession, presumably responses to 35, 36; sequence number is still 1 ... server retransmits, client sends same ACKs, etc. until server gets frustrated and sends a RST It looks like the MS FTP server is being well-behaved, but each time some data is received, the internal linux desktop immediately replies with an ACK denying that any data was received. This suggests that it's aware that it's received some packets, but is rejecting the contents for some reason. Not sure. Looking at the capture on firewall external interface, is pretty much the same, except that the second data packet (corresponding to #35) is completely missing. This might be a red herring. Like I said I supply this info just for curiosity. We have it working now by removing reassemble tcp - maybe its helpful to the PF developers, maybe not. Below is the screen output from the internal linux desktop using lftp. Many thanks, Alastair Johnson = NOT Working = -bash-3.1$ lftp -c open -uA,B 198.80.165.171; debug; ls Connecting to 198.80.165.171 (198.80.165.171) port 21 --- 220 Microsoft FTP Service --- FEAT --- 530 Please login with USER and PASS. --- AUTH TLS --- 500 'AUTH TLS': command not understood --- USER A --- 331 Password required for A. --- PASS B --- 230-Welcome to Thomson Financial I/B/E/S FTP site. --- This site is for authorized users only. --- Any or all uses of this system and all files on --- this system are monitored, recorded, and audited. --- LOG OFF IMMEDIATELY if you do not agree to the --- conditions stated in this warning. --- 230 User A logged in. --- FEAT --- 211-FEAT --- SIZE --- MDTM --- 211 END --- PWD --- 257 / is current directory. --- PASV --- 227 Entering Passive Mode (198,80,165,171,227,43) Connecting data socket to (198.80.165.171) port 58155 Data connection established --- LIST --- 125 Data connection already open; Transfer starting. --- 226 Transfer complete. `ls' at 0 [Receiving data] = Hang at this point == = Working = snip --- 230 User A logged in. --- FEAT --- 211-FEAT --- SIZE --- MDTM --- 211 END --- PWD --- 257 / is current directory. --- PASV --- 227 Entering Passive Mode (198,80,165,171,198,136) Connecting data socket to (198.80.165.171) port 50824 Data connection established --- LIST --- 125 Data connection already open; Transfer starting. dr-xr-xr-x 1 ownergroup 0 Dec 22 2009 history ... and then more files/directories
another reassemble tcp problem - details for PF developers
There has been 2 recent threads mentioning problems with reassemble tcp: pf: reassemble tcp problems with emails through pf For info here is another. We solved the problem by removing this scrub. We went from 4.3 with scrub in all to the latest 4.6 stable with match in all scrub (reassemble tcp) and bizarrely found we sometimes (~50%) couldnt FTP to several sites. It would hang just before sending any data. This would happen about 50% of the time and work fine the rest. It may be an irrelevant coincidence but each FTP site that this happened with was Microsoft FTP. It never happened (ie FTP always worked fine) with other server types. The OpenBSD firewall is running ftp-proxy. The internal client is a linux desktop. Removing the reassemble tcp scrub made the problem go away and a successful connection/download every time. We took tcpdumps on both the internal and external interfaces of the OpenBSD firewall/ftp-proxy. Looking at our capture on firewall internal interface, the payload of the first TCP packet after the handshake seems to be ignored. I see: Frames 26, 28, 29: TCP handshake 33: first data from server 34: client then immediately sends ACK to server with seq=1 35: last of the data from server 36: server attempting to close connection 37, 38: Client sends two ACKs in quick succession, presumably responses to 35, 36; sequence number is still 1 ... server retransmits, client sends same ACKs, etc. until server gets frustrated and sends a RST It looks like the MS FTP server is being well-behaved, but each time some data is received, the internal linux desktop immediately replies with an ACK denying that any data was received. This suggests that it's aware that it's received some packets, but is rejecting the contents for some reason. Not sure. Looking at the capture on firewall external interface, is pretty much the same, except that the second data packet (corresponding to #35) is completely missing. This might be a red herring. Like I said I supply this info just for curiosity. We have it working now by removing reassemble tcp - maybe its helpful to the PF developers, maybe not. Below is the screen output from the internal linux desktop using lftp. Many thanks, Alastair Johnson = NOT Working = -bash-3.1$ lftp -c open -uA,B 198.80.165.171; debug; ls Connecting to 198.80.165.171 (198.80.165.171) port 21 --- 220 Microsoft FTP Service --- FEAT --- 530 Please login with USER and PASS. --- AUTH TLS --- 500 'AUTH TLS': command not understood --- USER A --- 331 Password required for A. --- PASS B --- 230-Welcome to Thomson Financial I/B/E/S FTP site. --- This site is for authorized users only. --- Any or all uses of this system and all files on --- this system are monitored, recorded, and audited. --- LOG OFF IMMEDIATELY if you do not agree to the --- conditions stated in this warning. --- 230 User A logged in. --- FEAT --- 211-FEAT --- SIZE --- MDTM --- 211 END --- PWD --- 257 / is current directory. --- PASV --- 227 Entering Passive Mode (198,80,165,171,227,43) Connecting data socket to (198.80.165.171) port 58155 Data connection established --- LIST --- 125 Data connection already open; Transfer starting. --- 226 Transfer complete. `ls' at 0 [Receiving data] = Hang at this point == = Working = snip --- 230 User A logged in. --- FEAT --- 211-FEAT --- SIZE --- MDTM --- 211 END --- PWD --- 257 / is current directory. --- PASV --- 227 Entering Passive Mode (198,80,165,171,198,136) Connecting data socket to (198.80.165.171) port 50824 Data connection established --- LIST --- 125 Data connection already open; Transfer starting. dr-xr-xr-x 1 owner group 0 Dec 22 2009 history ... and then more files/directories
Re: another reassemble tcp problem - details for PF developers
On 2010-01-15, Alastair Johnson att...@googlemail.com wrote: We went from 4.3 with scrub in all to the latest 4.6 stable with match in all scrub (reassemble tcp) 'scrub in all' on 4.3 is the same as 'scrub in all fragment reassemble' which is now done by default. If you had done 'scrub in all reassemble tcp' on 4.3 you would probably see the problem there too. and bizarrely found we sometimes (~50%) couldn?t FTP to several sites. Do you have any examples of public sites affected? ftp.microsoft.com doesn't do this for me via 'scrub (reassemble tcp)' in -current.
Re: another reassemble tcp problem - details for PF developers
no - only ftp to ftp.ibes.com and ftp to www.chi-x.com which are not public. it seems to have failed after login just before or just as data started to flow. the strange thing is that data download works about half the time with reassemble tcp switched on and all the time with it off. appologies if i sent to the list more than once. could possibly provide a traffic capture but i'm not sure how to strip out unencrypted login details. 2010/1/15 Stuart Henderson s...@spacehopper.org On 2010-01-15, Alastair Johnson att...@googlemail.com wrote: We went from 4.3 with scrub in all to the latest 4.6 stable with match in all scrub (reassemble tcp) 'scrub in all' on 4.3 is the same as 'scrub in all fragment reassemble' which is now done by default. If you had done 'scrub in all reassemble tcp' on 4.3 you would probably see the problem there too. and bizarrely found we sometimes (~50%) couldn?t FTP to several sites. Do you have any examples of public sites affected? ftp.microsoft.com doesn't do this for me via 'scrub (reassemble tcp)' in -current.
Re: another reassemble tcp problem - details for PF developers
snip It may be an irrelevant coincidence but each FTP site that this happened with was Microsoft FTP. It never happened (ie FTP always worked fine) with other server types. snip I don't know if this matters, but I had some problems recently with people downloading from external FTP servers and we found it had something to do with the FTP client and the use of either the client of external server using passive mode. I can't remember the exact details but it came down to a the ms-client/server working differently then expected...this probably won't help anyone but it might
another reassemble tcp problem - details for PF developers
There has been 2 recent threads mentioning problems with reassemble tcp: pf: reassemble tcp problems with emails through pf For info here is another. We solved the problem by removing this scrub. We went from 4.3 with scrub in all to the latest 4.6 stable with match in all scrub (reassemble tcp) and bizarrely found we sometimes (~50%) couldnt FTP to several sites. It would hang just before sending any data. This would happen about 50% of the time and work fine the rest. It may be an irrelevant coincidence but each FTP site that this happened with was Microsoft FTP. It never happened (ie FTP always worked fine) with other server types. The OpenBSD firewall is running ftp-proxy. The internal client is a linux desktop. Removing the reassemble tcp scrub made the problem go away and a successful connection/download every time. We took tcpdumps on both the internal and external interfaces of the OpenBSD firewall/ftp-proxy. Looking at our capture on firewall internal interface, the payload of the first TCP packet after the handshake seems to be ignored. I see: Frames 26, 28, 29: TCP handshake 33: first data from server 34: client then immediately sends ACK to server with seq=1 35: last of the data from server 36: server attempting to close connection 37, 38: Client sends two ACKs in quick succession, presumably responses to 35, 36; sequence number is still 1 ... server retransmits, client sends same ACKs, etc. until server gets frustrated and sends a RST It looks like the MS FTP server is being well-behaved, but each time some data is received, the internal linux desktop immediately replies with an ACK denying that any data was received. This suggests that it's aware that it's received some packets, but is rejecting the contents for some reason. Not sure. Looking at the capture on firewall external interface, is pretty much the same, except that the second data packet (corresponding to #35) is completely missing. This might be a red herring. Like I said I supply this info just for curiosity. We have it working now by removing reassemble tcp - maybe its helpful to the PF developers, maybe not. Below is the screen output from the internal linux desktop using lftp. Many thanks, Alastair Johnson = NOT Working = -bash-3.1$ lftp -c open -uA,B 198.80.165.171; debug; ls Connecting to 198.80.165.171 (198.80.165.171) port 21 --- 220 Microsoft FTP Service --- FEAT --- 530 Please login with USER and PASS. --- AUTH TLS --- 500 'AUTH TLS': command not understood --- USER A --- 331 Password required for A. --- PASS B --- 230-Welcome to Thomson Financial I/B/E/S FTP site. --- This site is for authorized users only. --- Any or all uses of this system and all files on --- this system are monitored, recorded, and audited. --- LOG OFF IMMEDIATELY if you do not agree to the --- conditions stated in this warning. --- 230 User A logged in. --- FEAT --- 211-FEAT --- SIZE --- MDTM --- 211 END --- PWD --- 257 / is current directory. --- PASV --- 227 Entering Passive Mode (198,80,165,171,227,43) Connecting data socket to (198.80.165.171) port 58155 Data connection established --- LIST --- 125 Data connection already open; Transfer starting. --- 226 Transfer complete. `ls' at 0 [Receiving data] = Hang at this point == = Working = snip --- 230 User A logged in. --- FEAT --- 211-FEAT --- SIZE --- MDTM --- 211 END --- PWD --- 257 / is current directory. --- PASV --- 227 Entering Passive Mode (198,80,165,171,198,136) Connecting data socket to (198.80.165.171) port 50824 Data connection established --- LIST --- 125 Data connection already open; Transfer starting. dr-xr-xr-x 1 ownergroup 0 Dec 22 2009 history ... and then more files/directories
reassemble tcp problem - details for PF developers
There has been 2 recent threads mentioning problems with reassemble tcp: pf: reassemble tcp problems with emails through pf For info here is another. We solved the problem by removing this scrub. We went from 4.3 with scrub in all to the latest 4.6 stable with match in all scrub (reassemble tcp) and bizarrely found we sometimes (~50%) couldnt FTP to several sites. It would hang just before sending any data. This would happen about 50% of the time and work fine the rest. It may be an irrelevant coincidence but each FTP site that this happened with was Microsoft FTP. It never happened (ie FTP always worked fine) with other server types. The OpenBSD firewall is running ftp-proxy. The internal client is a linux desktop. Removing the reassemble tcp scrub made the problem go away and a successful connection/download every time. We took tcpdumps on both the internal and external interfaces of the OpenBSD firewall/ftp-proxy. Looking at our capture on firewall internal interface, the payload of the first TCP packet after the handshake seems to be ignored. I see: Frames 26, 28, 29: TCP handshake 33: first data from server 34: client then immediately sends ACK to server with seq=1 35: last of the data from server 36: server attempting to close connection 37, 38: Client sends two ACKs in quick succession, presumably responses to 35, 36; sequence number is still 1 ... server retransmits, client sends same ACKs, etc. until server gets frustrated and sends a RST It looks like the MS FTP server is being well-behaved, but each time some data is received, the internal linux desktop immediately replies with an ACK denying that any data was received. This suggests that it's aware that it's received some packets, but is rejecting the contents for some reason. Not sure. Looking at the capture on firewall external interface, is pretty much the same, except that the second data packet (corresponding to #35) is completely missing. This might be a red herring. Like I said I supply this info just for curiosity. We have it working now by removing reassemble tcp - maybe its helpful to the PF developers, maybe not. Below is the screen output from the internal linux desktop using lftp. Many thanks, Alastair Johnson = NOT Working = -bash-3.1$ lftp -c open -uA,B 198.80.165.171; debug; ls Connecting to 198.80.165.171 (198.80.165.171) port 21 --- 220 Microsoft FTP Service --- FEAT --- 530 Please login with USER and PASS. --- AUTH TLS --- 500 'AUTH TLS': command not understood --- USER A --- 331 Password required for A. --- PASS B --- 230-Welcome to Thomson Financial I/B/E/S FTP site. --- This site is for authorized users only. --- Any or all uses of this system and all files on --- this system are monitored, recorded, and audited. --- LOG OFF IMMEDIATELY if you do not agree to the --- conditions stated in this warning. --- 230 User A logged in. --- FEAT --- 211-FEAT --- SIZE --- MDTM --- 211 END --- PWD --- 257 / is current directory. --- PASV --- 227 Entering Passive Mode (198,80,165,171,227,43) Connecting data socket to (198.80.165.171) port 58155 Data connection established --- LIST --- 125 Data connection already open; Transfer starting. --- 226 Transfer complete. `ls' at 0 [Receiving data] = Hang at this point == = Working = snip --- 230 User A logged in. --- FEAT --- 211-FEAT --- SIZE --- MDTM --- 211 END --- PWD --- 257 / is current directory. --- PASV --- 227 Entering Passive Mode (198,80,165,171,198,136) Connecting data socket to (198.80.165.171) port 50824 Data connection established --- LIST --- 125 Data connection already open; Transfer starting. dr-xr-xr-x 1 ownergroup 0 Dec 22 2009 history ... and then more files/directories
