Re: pf: reassemble tcp
* Sonic sonicsm...@gmail.com [2014-09-05 17:12]: On Fri, Sep 5, 2014 at 4:42 AM, Kapetanakis Giannis bil...@edu.physics.uoc.gr wrote: yeah, don't use reassemble tcp. it's not perfect. Isn't that default behavior? hell, no. -- Henning Brauer, h...@bsws.de, henn...@openbsd.org BS Web Services GmbH, http://bsws.de, Full-Service ISP Secure Hosting, Mail and DNS. Virtual Dedicated Servers, Root to Fully Managed Henning Brauer Consulting, http://henningbrauer.com/
Re: pf: reassemble tcp
* Kapetanakis Giannis bil...@edu.physics.uoc.gr [2014-09-06 00:50]: I'm asking about reassemble tcp. According to some 2010's threads in misc@ it used to cause problems to some users. I'm wondering what's the status now. unchanged. -- Henning Brauer, h...@bsws.de, henn...@openbsd.org BS Web Services GmbH, http://bsws.de, Full-Service ISP Secure Hosting, Mail and DNS. Virtual Dedicated Servers, Root to Fully Managed Henning Brauer Consulting, http://henningbrauer.com/
Re: pf: reassemble tcp
On 13/09/14 11:55, Henning Brauer wrote: * Kapetanakis Giannis bil...@edu.physics.uoc.gr [2014-09-06 00:50]: I'm asking about reassemble tcp. According to some 2010's threads in misc@ it used to cause problems to some users. I'm wondering what's the status now. unchanged. Thanks for the reply G
Re: pf: reassemble tcp
I've found the following in the archives. Is the situation still the same with reassemble tcp? My only scrub rule (in firewall/router) is match in all scrub (no-df random-id reassemble tcp max-mss 1440) Should I be worried? Thanks G List: openbsd-misc Subject:Re: pf: reassemble tcp From: Henning Brauer lists-openbsd () bsws ! de Date: 2010-01-14 1:46:17 Message-ID: 20100114014617.GH3135 () nudo ! bsws ! de [Download message RAW] * nixlists nixmli...@gmail.com [2010-01-13 22:56]: Hi. I have match in all scrub (tcp reassemble no-df random-id max-mss 1440) in my pf.conf (-current) Unless I remove 'tcp reassemble', one of the web sites (it's a Windows/IIS) site cannot communicate with me - it hangs loading a page. Any ideas? yeah, don't use reassemble tcp. it's not perfect. -- Henning Brauer, h...@bsws.de, henn...@openbsd.org BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting
Re: pf: reassemble tcp
On Fri, Sep 5, 2014 at 4:42 AM, Kapetanakis Giannis bil...@edu.physics.uoc.gr wrote: yeah, don't use reassemble tcp. it's not perfect. Isn't that default behavior? Is it recommended to disable this feature?
Re: pf: reassemble tcp
On 05/09/14 18:10, Sonic wrote: On Fri, Sep 5, 2014 at 4:42 AM, Kapetanakis Giannis bil...@edu.physics.uoc.gr wrote: yeah, don't use reassemble tcp. it's not perfect. Isn't that default behavior? Is it recommended to disable this feature? I'm not asking about set reassemble for fragmented packets (which in on by default), I'm asking about reassemble tcp. According to some 2010's threads in misc@ it used to cause problems to some users. I'm wondering what's the status now. regards, G
another reassemble tcp problem - details for PF developers
There has been 2 recent threads mentioning problems with reassemble tcp: pf: reassemble tcp problems with emails through pf For info here is another. We solved the problem by removing this scrub. We went from 4.3 with scrub in all to the latest 4.6 stable with match in all scrub (reassemble tcp) and bizarrely found we sometimes (~50%) couldnt FTP to several sites. It would hang just before sending any data. This would happen about 50% of the time and work fine the rest. It may be an irrelevant coincidence but each FTP site that this happened with was Microsoft FTP. It never happened (ie FTP always worked fine) with other server types. The OpenBSD firewall is running ftp-proxy. The internal client is a linux desktop. Removing the reassemble tcp scrub made the problem go away and a successful connection/download every time. We took tcpdumps on both the internal and external interfaces of the OpenBSD firewall/ftp-proxy. Looking at our capture on firewall internal interface, the payload of the first TCP packet after the handshake seems to be ignored. I see: Frames 26, 28, 29: TCP handshake 33: first data from server 34: client then immediately sends ACK to server with seq=1 35: last of the data from server 36: server attempting to close connection 37, 38: Client sends two ACKs in quick succession, presumably responses to 35, 36; sequence number is still 1 ... server retransmits, client sends same ACKs, etc. until server gets frustrated and sends a RST It looks like the MS FTP server is being well-behaved, but each time some data is received, the internal linux desktop immediately replies with an ACK denying that any data was received. This suggests that it's aware that it's received some packets, but is rejecting the contents for some reason. Not sure. Looking at the capture on firewall external interface, is pretty much the same, except that the second data packet (corresponding to #35) is completely missing. This might be a red herring. Like I said I supply this info just for curiosity. We have it working now by removing reassemble tcp - maybe its helpful to the PF developers, maybe not. Below is the screen output from the internal linux desktop using lftp. Many thanks, Alastair Johnson = NOT Working = -bash-3.1$ lftp -c open -uA,B 198.80.165.171; debug; ls Connecting to 198.80.165.171 (198.80.165.171) port 21 --- 220 Microsoft FTP Service --- FEAT --- 530 Please login with USER and PASS. --- AUTH TLS --- 500 'AUTH TLS': command not understood --- USER A --- 331 Password required for A. --- PASS B --- 230-Welcome to Thomson Financial I/B/E/S FTP site. --- This site is for authorized users only. --- Any or all uses of this system and all files on --- this system are monitored, recorded, and audited. --- LOG OFF IMMEDIATELY if you do not agree to the --- conditions stated in this warning. --- 230 User A logged in. --- FEAT --- 211-FEAT --- SIZE --- MDTM --- 211 END --- PWD --- 257 / is current directory. --- PASV --- 227 Entering Passive Mode (198,80,165,171,227,43) Connecting data socket to (198.80.165.171) port 58155 Data connection established --- LIST --- 125 Data connection already open; Transfer starting. --- 226 Transfer complete. `ls' at 0 [Receiving data] = Hang at this point == = Working = snip --- 230 User A logged in. --- FEAT --- 211-FEAT --- SIZE --- MDTM --- 211 END --- PWD --- 257 / is current directory. --- PASV --- 227 Entering Passive Mode (198,80,165,171,198,136) Connecting data socket to (198.80.165.171) port 50824 Data connection established --- LIST --- 125 Data connection already open; Transfer starting. dr-xr-xr-x 1 ownergroup 0 Dec 22 2009 history ... and then more files/directories
another reassemble tcp problem - details for PF developers
There has been 2 recent threads mentioning problems with reassemble tcp: pf: reassemble tcp problems with emails through pf For info here is another. We solved the problem by removing this scrub. We went from 4.3 with scrub in all to the latest 4.6 stable with match in all scrub (reassemble tcp) and bizarrely found we sometimes (~50%) couldnt FTP to several sites. It would hang just before sending any data. This would happen about 50% of the time and work fine the rest. It may be an irrelevant coincidence but each FTP site that this happened with was Microsoft FTP. It never happened (ie FTP always worked fine) with other server types. The OpenBSD firewall is running ftp-proxy. The internal client is a linux desktop. Removing the reassemble tcp scrub made the problem go away and a successful connection/download every time. We took tcpdumps on both the internal and external interfaces of the OpenBSD firewall/ftp-proxy. Looking at our capture on firewall internal interface, the payload of the first TCP packet after the handshake seems to be ignored. I see: Frames 26, 28, 29: TCP handshake 33: first data from server 34: client then immediately sends ACK to server with seq=1 35: last of the data from server 36: server attempting to close connection 37, 38: Client sends two ACKs in quick succession, presumably responses to 35, 36; sequence number is still 1 ... server retransmits, client sends same ACKs, etc. until server gets frustrated and sends a RST It looks like the MS FTP server is being well-behaved, but each time some data is received, the internal linux desktop immediately replies with an ACK denying that any data was received. This suggests that it's aware that it's received some packets, but is rejecting the contents for some reason. Not sure. Looking at the capture on firewall external interface, is pretty much the same, except that the second data packet (corresponding to #35) is completely missing. This might be a red herring. Like I said I supply this info just for curiosity. We have it working now by removing reassemble tcp - maybe its helpful to the PF developers, maybe not. Below is the screen output from the internal linux desktop using lftp. Many thanks, Alastair Johnson = NOT Working = -bash-3.1$ lftp -c open -uA,B 198.80.165.171; debug; ls Connecting to 198.80.165.171 (198.80.165.171) port 21 --- 220 Microsoft FTP Service --- FEAT --- 530 Please login with USER and PASS. --- AUTH TLS --- 500 'AUTH TLS': command not understood --- USER A --- 331 Password required for A. --- PASS B --- 230-Welcome to Thomson Financial I/B/E/S FTP site. --- This site is for authorized users only. --- Any or all uses of this system and all files on --- this system are monitored, recorded, and audited. --- LOG OFF IMMEDIATELY if you do not agree to the --- conditions stated in this warning. --- 230 User A logged in. --- FEAT --- 211-FEAT --- SIZE --- MDTM --- 211 END --- PWD --- 257 / is current directory. --- PASV --- 227 Entering Passive Mode (198,80,165,171,227,43) Connecting data socket to (198.80.165.171) port 58155 Data connection established --- LIST --- 125 Data connection already open; Transfer starting. --- 226 Transfer complete. `ls' at 0 [Receiving data] = Hang at this point == = Working = snip --- 230 User A logged in. --- FEAT --- 211-FEAT --- SIZE --- MDTM --- 211 END --- PWD --- 257 / is current directory. --- PASV --- 227 Entering Passive Mode (198,80,165,171,198,136) Connecting data socket to (198.80.165.171) port 50824 Data connection established --- LIST --- 125 Data connection already open; Transfer starting. dr-xr-xr-x 1 owner group 0 Dec 22 2009 history ... and then more files/directories
Re: another reassemble tcp problem - details for PF developers
On 2010-01-15, Alastair Johnson att...@googlemail.com wrote: We went from 4.3 with scrub in all to the latest 4.6 stable with match in all scrub (reassemble tcp) 'scrub in all' on 4.3 is the same as 'scrub in all fragment reassemble' which is now done by default. If you had done 'scrub in all reassemble tcp' on 4.3 you would probably see the problem there too. and bizarrely found we sometimes (~50%) couldn?t FTP to several sites. Do you have any examples of public sites affected? ftp.microsoft.com doesn't do this for me via 'scrub (reassemble tcp)' in -current.
Re: another reassemble tcp problem - details for PF developers
no - only ftp to ftp.ibes.com and ftp to www.chi-x.com which are not public. it seems to have failed after login just before or just as data started to flow. the strange thing is that data download works about half the time with reassemble tcp switched on and all the time with it off. appologies if i sent to the list more than once. could possibly provide a traffic capture but i'm not sure how to strip out unencrypted login details. 2010/1/15 Stuart Henderson s...@spacehopper.org On 2010-01-15, Alastair Johnson att...@googlemail.com wrote: We went from 4.3 with scrub in all to the latest 4.6 stable with match in all scrub (reassemble tcp) 'scrub in all' on 4.3 is the same as 'scrub in all fragment reassemble' which is now done by default. If you had done 'scrub in all reassemble tcp' on 4.3 you would probably see the problem there too. and bizarrely found we sometimes (~50%) couldn?t FTP to several sites. Do you have any examples of public sites affected? ftp.microsoft.com doesn't do this for me via 'scrub (reassemble tcp)' in -current.
Re: another reassemble tcp problem - details for PF developers
snip It may be an irrelevant coincidence but each FTP site that this happened with was Microsoft FTP. It never happened (ie FTP always worked fine) with other server types. snip I don't know if this matters, but I had some problems recently with people downloading from external FTP servers and we found it had something to do with the FTP client and the use of either the client of external server using passive mode. I can't remember the exact details but it came down to a the ms-client/server working differently then expected...this probably won't help anyone but it might
reassemble tcp
Hi there. I've a problem with pf on OpenBSD 4.6 After different test, I've been reduced my pf.conf to those rules: macros set block-policy drop match all scrub (no-df, random-id, reassemble tcp, max-mss 1440) nat on $ext from $int:network - $ext:0 block log all pass in on $int from any to any pass out on $ext from $ext:0 to any pfctl get all rules without errors, but I've problem during connection. If I try to get login with pidgin (MSN) from slackware Linux It doesn't work. If I try to get login with pidgin proxied from slackware it works. I've tried also to remove reassemble tcp from the scrub and it works If I try to get login with MSN from windows (proxied, with reassemble tcp, and no proxy) It works. In all Linux pidgin failed connection I receive this: connection: Connection error on 0x8551180 (reason: 0 description: Connection error from Notification server: Reading error) But the connection will be dropped? (I receive also a block log of ack for the pidgin connection) Another problem with reassemble tcp is with windows boot. I receive from syslog those messages: block in on rl0: 10.1.3.53.137 10.1.255.255.137: udp 50 If I remove reassemble tcp It works fine. I've tried also with a pass all rules...but with the same result. It's possible that a scrub with reassemble tcp option, blocks some packet? What is the reason for this? It's a my misconfiguration or is a normal behaviuour? Thanks in advance!
Re: reassemble tcp
On Fri, Jan 15, 2010 at 3:33 PM, Alessandro Baggi alessandro.ba...@gmail.com wrote: If I remove reassemble tcp It works fine. I've tried also with a pass all rules...but with the same result. It's possible that a scrub with reassemble tcp option, blocks some packet? What is the reason for this? http://marc.info/?l=openbsd-miscm=126344466917828w=2
Re: reassemble tcp
Ted Unangst wrote: On Fri, Jan 15, 2010 at 3:33 PM, Alessandro Baggi alessandro.ba...@gmail.com wrote: If I remove reassemble tcp It works fine. I've tried also with a pass all rules...but with the same result. It's possible that a scrub with reassemble tcp option, blocks some packet? What is the reason for this? http://marc.info/?l=openbsd-miscm=126344466917828w=2 Hi ted, thanks for the reply. but then what's the meaning of this options?
another reassemble tcp problem - details for PF developers
There has been 2 recent threads mentioning problems with reassemble tcp: pf: reassemble tcp problems with emails through pf For info here is another. We solved the problem by removing this scrub. We went from 4.3 with scrub in all to the latest 4.6 stable with match in all scrub (reassemble tcp) and bizarrely found we sometimes (~50%) couldnt FTP to several sites. It would hang just before sending any data. This would happen about 50% of the time and work fine the rest. It may be an irrelevant coincidence but each FTP site that this happened with was Microsoft FTP. It never happened (ie FTP always worked fine) with other server types. The OpenBSD firewall is running ftp-proxy. The internal client is a linux desktop. Removing the reassemble tcp scrub made the problem go away and a successful connection/download every time. We took tcpdumps on both the internal and external interfaces of the OpenBSD firewall/ftp-proxy. Looking at our capture on firewall internal interface, the payload of the first TCP packet after the handshake seems to be ignored. I see: Frames 26, 28, 29: TCP handshake 33: first data from server 34: client then immediately sends ACK to server with seq=1 35: last of the data from server 36: server attempting to close connection 37, 38: Client sends two ACKs in quick succession, presumably responses to 35, 36; sequence number is still 1 ... server retransmits, client sends same ACKs, etc. until server gets frustrated and sends a RST It looks like the MS FTP server is being well-behaved, but each time some data is received, the internal linux desktop immediately replies with an ACK denying that any data was received. This suggests that it's aware that it's received some packets, but is rejecting the contents for some reason. Not sure. Looking at the capture on firewall external interface, is pretty much the same, except that the second data packet (corresponding to #35) is completely missing. This might be a red herring. Like I said I supply this info just for curiosity. We have it working now by removing reassemble tcp - maybe its helpful to the PF developers, maybe not. Below is the screen output from the internal linux desktop using lftp. Many thanks, Alastair Johnson = NOT Working = -bash-3.1$ lftp -c open -uA,B 198.80.165.171; debug; ls Connecting to 198.80.165.171 (198.80.165.171) port 21 --- 220 Microsoft FTP Service --- FEAT --- 530 Please login with USER and PASS. --- AUTH TLS --- 500 'AUTH TLS': command not understood --- USER A --- 331 Password required for A. --- PASS B --- 230-Welcome to Thomson Financial I/B/E/S FTP site. --- This site is for authorized users only. --- Any or all uses of this system and all files on --- this system are monitored, recorded, and audited. --- LOG OFF IMMEDIATELY if you do not agree to the --- conditions stated in this warning. --- 230 User A logged in. --- FEAT --- 211-FEAT --- SIZE --- MDTM --- 211 END --- PWD --- 257 / is current directory. --- PASV --- 227 Entering Passive Mode (198,80,165,171,227,43) Connecting data socket to (198.80.165.171) port 58155 Data connection established --- LIST --- 125 Data connection already open; Transfer starting. --- 226 Transfer complete. `ls' at 0 [Receiving data] = Hang at this point == = Working = snip --- 230 User A logged in. --- FEAT --- 211-FEAT --- SIZE --- MDTM --- 211 END --- PWD --- 257 / is current directory. --- PASV --- 227 Entering Passive Mode (198,80,165,171,198,136) Connecting data socket to (198.80.165.171) port 50824 Data connection established --- LIST --- 125 Data connection already open; Transfer starting. dr-xr-xr-x 1 ownergroup 0 Dec 22 2009 history ... and then more files/directories
reassemble tcp problem - details for PF developers
There has been 2 recent threads mentioning problems with reassemble tcp: pf: reassemble tcp problems with emails through pf For info here is another. We solved the problem by removing this scrub. We went from 4.3 with scrub in all to the latest 4.6 stable with match in all scrub (reassemble tcp) and bizarrely found we sometimes (~50%) couldnt FTP to several sites. It would hang just before sending any data. This would happen about 50% of the time and work fine the rest. It may be an irrelevant coincidence but each FTP site that this happened with was Microsoft FTP. It never happened (ie FTP always worked fine) with other server types. The OpenBSD firewall is running ftp-proxy. The internal client is a linux desktop. Removing the reassemble tcp scrub made the problem go away and a successful connection/download every time. We took tcpdumps on both the internal and external interfaces of the OpenBSD firewall/ftp-proxy. Looking at our capture on firewall internal interface, the payload of the first TCP packet after the handshake seems to be ignored. I see: Frames 26, 28, 29: TCP handshake 33: first data from server 34: client then immediately sends ACK to server with seq=1 35: last of the data from server 36: server attempting to close connection 37, 38: Client sends two ACKs in quick succession, presumably responses to 35, 36; sequence number is still 1 ... server retransmits, client sends same ACKs, etc. until server gets frustrated and sends a RST It looks like the MS FTP server is being well-behaved, but each time some data is received, the internal linux desktop immediately replies with an ACK denying that any data was received. This suggests that it's aware that it's received some packets, but is rejecting the contents for some reason. Not sure. Looking at the capture on firewall external interface, is pretty much the same, except that the second data packet (corresponding to #35) is completely missing. This might be a red herring. Like I said I supply this info just for curiosity. We have it working now by removing reassemble tcp - maybe its helpful to the PF developers, maybe not. Below is the screen output from the internal linux desktop using lftp. Many thanks, Alastair Johnson = NOT Working = -bash-3.1$ lftp -c open -uA,B 198.80.165.171; debug; ls Connecting to 198.80.165.171 (198.80.165.171) port 21 --- 220 Microsoft FTP Service --- FEAT --- 530 Please login with USER and PASS. --- AUTH TLS --- 500 'AUTH TLS': command not understood --- USER A --- 331 Password required for A. --- PASS B --- 230-Welcome to Thomson Financial I/B/E/S FTP site. --- This site is for authorized users only. --- Any or all uses of this system and all files on --- this system are monitored, recorded, and audited. --- LOG OFF IMMEDIATELY if you do not agree to the --- conditions stated in this warning. --- 230 User A logged in. --- FEAT --- 211-FEAT --- SIZE --- MDTM --- 211 END --- PWD --- 257 / is current directory. --- PASV --- 227 Entering Passive Mode (198,80,165,171,227,43) Connecting data socket to (198.80.165.171) port 58155 Data connection established --- LIST --- 125 Data connection already open; Transfer starting. --- 226 Transfer complete. `ls' at 0 [Receiving data] = Hang at this point == = Working = snip --- 230 User A logged in. --- FEAT --- 211-FEAT --- SIZE --- MDTM --- 211 END --- PWD --- 257 / is current directory. --- PASV --- 227 Entering Passive Mode (198,80,165,171,198,136) Connecting data socket to (198.80.165.171) port 50824 Data connection established --- LIST --- 125 Data connection already open; Transfer starting. dr-xr-xr-x 1 ownergroup 0 Dec 22 2009 history ... and then more files/directories
pf: reassemble tcp
Hi. I have match in all scrub (tcp reassemble no-df random-id max-mss 1440) in my pf.conf (-current) Unless I remove 'tcp reassemble', one of the web sites (it's a Windows/IIS) site cannot communicate with me - it hangs loading a page. Any ideas?
Re: pf: reassemble tcp
* nixlists nixmli...@gmail.com [2010-01-13 22:56]: Hi. I have match in all scrub (tcp reassemble no-df random-id max-mss 1440) in my pf.conf (-current) Unless I remove 'tcp reassemble', one of the web sites (it's a Windows/IIS) site cannot communicate with me - it hangs loading a page. Any ideas? yeah, don't use reassemble tcp. it's not perfect. -- Henning Brauer, h...@bsws.de, henn...@openbsd.org BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting
Re: pf: reassemble tcp
On Thu, Jan 14, 2010 at 12:46 PM, Henning Brauer lists-open...@bsws.dewrote: I have match in all scrub (tcp reassemble no-df random-id max-mss 1440) in my pf.conf (-current) yeah, don't use reassemble tcp. it's not perfect. How about fragment reassemble? I'm using it on my OpenBSD 4.5 pf, with scrub to enable a NAT AV app to work. Reading the man pages I noticed fragment reassemble has changed to set reassembleunder scrub for 4.6 or -current. It also looks like it is turned on by default in 4.5, 4.6 or current.
Re: pf: reassemble tcp
* Ted t...@pobox.com [2010-01-14 05:03]: On Thu, Jan 14, 2010 at 12:46 PM, Henning Brauer lists-open...@bsws.dewrote: I have match in all scrub (tcp reassemble no-df random-id max-mss 1440) in my pf.conf (-current) yeah, don't use reassemble tcp. it's not perfect. How about fragment reassemble? that is an entirely different beast and should always be on (hey, surprise, it IS by default!) reassemble tcp is not the best name really. it is not really reassembly of anything. -- Henning Brauer, h...@bsws.de, henn...@openbsd.org BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting
Re: scrub reassemble tcp and nat causes problems with some sites
Get tcpdumps on both router interfaces with and without the reassemble tcp option. Do this for a similar file on both a working website and broken (ebay) website. On both router interfaces? Wouldn't the external if be enough? You're probably right. But my theory is that if you're going to go to the effort of getting some comparable captures you may as well get them all at the same time. Having both external and internal is helpful for debugging issues with your router itself - i.e. it would indicate if the packets arrived looking good and then, after pf, looked bad.
Re: scrub reassemble tcp and nat causes problems with some sitesB
On Fri, 21 Jul 2006, Daniel E. Hassler wrote: Yes. I called it a Transparent Packet Filter (TPF) - the OpenBSD system is acting as a bridge. It's transparent because neither of the interfaces has an IP configured. WAN---PIX---DMZ---TPF---LAN---OS X Oh yes, I recall that image from one of your posts. It's a filtering bridge then. Would be interesting if bridging has something to do with our reassemble tcp problem... Perhaps because the if is in promiscuous mode? However, as I've mentioned earlier, everything works _on_ the gateway _without_ NAT. But regardless of NAT, all packets cross the bridge which connects the OpenBSD VM to the external gateway (cablemodem). Walter PS: I'll do a cc to the mailing list too to let people know we're having a bridge involved.
Re: scrub reassemble tcp and nat causes problems with some sites
On Thu, 20 Jul 2006, Steve Welham wrote: Get tcpdumps on both router interfaces with and without the reassemble tcp option. Do this for a similar file on both a working website and broken (ebay) website. I have now. Got a dump of the following request (all on a single line): wget -nd -O /dev/null -Y off http://de.ebayobjects.com/6k;h=v7/3429/0/0/%2a/b;32478602;0-0;0;8693398;13191-275/73;16963851/16981746/1;;~sscs=%3fhttp://listings.ebay.de/_W0QQsocmdZListingItemList?sofocus=sosbrftog=1catref=C3fccl=1from=R2fcl=4socmd=ListingItemListsatitle=sacat=25863%26catref%3DC6fsop=1%26fsoo%3D1fgtp=a6=-24a56=-24a51=-24a45419=-24gcs=1884pfid=2674reqtype=2pfmode=1alist=a6%2Ca56%2Ca51%2Ca3801%2Ca45419pf_query=sargn=-1%26saslc%3D3sascs=2ga10244=10425saslt=2saprclo=saprchi=so=Artikel+anzeigen; This is just an URL of one of the ad images displayed on the right hand side on ebay.de. With reassemble tcp I get reproducible timeouts. Then load the comparable captures into Ethereal/Wireshark and stare at them until it makes sense :-) Well, both dumps of working and timing out requests are really quite similiar with two notable exceptions: With reassemble tcp, the received HTTP response packets (HTTP 1.1 ...) have a BAD packet checksum! However, this is not true all the time, just sometimes. Would have been to easy anyways... Interestingly though, the delay between the last packet sent (usually an ACK) and the next packet received (HTTP continuation) is most of the time surprisingly long, ranging from tenths of a second, a couple of seconds and upto over a minute and even longer when wget finally times out with a read error. Without reassemble tcp, there are no checksum errors and no long delays, everything is usually well below 10 ms then. Can somebody draw some conclusions from this behaviour? Or at least have a good guess? ;-) I've also tried 'set debug load' if pf.conf because of Daniel's suggestion in this thread. Occasionally I get a loose state match but the syslog timestamps do not really correlate with the dump timestamps (all ntp synced). I've saved all the tcpdumps and the wget logs in case somebody wants to examine them. Just tell me and I'll email them. Walter
Re: scrub reassemble tcp and nat causes problems with some sitesB
On Thu, 20 Jul 2006, Daniel E. Hassler wrote: I was hoping he could try 'set debug loud' in his pf.conf and check his /var/log/messages file after testing a problem site. If he sees messages similar to the one's I've seen maybe we both know a little more. Unfortunately not. When I set debug to load, I just get some (not many) 'loose state match' messages in syslog. However, not only for the timing out sites, but also for e.g. pop3 sessions or else. Don't seem to cause any troubles as far as I can tell. Walter
Re: scrub reassemble tcp and nat causes problems with some sites
Sorry, 'modulate tcp' was a thinko. I had been meaning to move 'modulate state' into the scrubber for a long time. Reassemble TCP does aggressive TCP PAWs checks on the TCP timestamps. It does the usual PAWs check to make sure a timestamp is not older than the last echoed value - which is in theory a wrapped sequence number. It also does its aggressive check to make sure the timestamp did not increase faster than the fastest clock which the RFC allows - an attacker can kill a connection by spoofing a higher timestamp. In order to prevent blind TCP spoofing PF's scrub will require TCP timestamps on all data packets if the first data packet had a timestamp. There are some transparent web caches out there that will allow the 3whs to complete (exchanging timestamp information) and then they will will hijack the TCP connection. Of course the developers of the hijacking device took some liberal shortcuts that make them look just like a blind hijacker instead of a MITM hijacker. If they hijack in the middle of the data stream (like after they see the HTTP request) and stop sending TCP Timestamps then PF's reassemble TCP will block further packets. .mike What is 'modulate tcp'? modulate state works fine. I get these errors only with scrub's reassemble tcp option I originally assumed it was an Apple problem since I only had trouble with the OS X Software Update feature. Going back to the beginning of this thread - Walter Haidinger appears to have a similar problem but not with Apple. I was hoping he could try 'set debug loud' in his pf.conf and check his /var/log/messages file after testing a problem site. If he sees messages similar to the one's I've seen maybe we both know a little more. -Dan
Re: scrub reassemble tcp and nat causes problems with some sites
On Fri, 21 Jul 2006, Mike Frantzen wrote: Reassemble TCP does aggressive TCP PAWs checks on the TCP timestamps. It does the usual PAWs check to make sure a timestamp is not older than the last echoed value - which is in theory a wrapped sequence number. It also does its aggressive check to make sure the timestamp did not increase faster than the fastest clock which the RFC allows - an attacker can kill a connection by spoofing a higher timestamp. In order to prevent blind TCP spoofing PF's scrub will require TCP timestamps on all data packets if the first data packet had a timestamp. There are some transparent web caches out there that will allow the 3whs to complete (exchanging timestamp information) and then they will will hijack the TCP connection. Of course the developers of the hijacking device took some liberal shortcuts that make them look just like a blind hijacker instead of a MITM hijacker. If they hijack in the middle of the data stream (like after they see the HTTP request) and stop sending TCP Timestamps then PF's reassemble TCP will block further packets. How does this interfere with the NAT code? Reassemble tcp has _only_ problems from a machine behind the OpenBSD NAT router. On the router itself or if using a proxy running on it, there is no problem. So I'd think that it is actually an OpenBSD bug in the NAT and/or reassemble tcp code. How can this be verified? Regards, Walter
Re: scrub reassemble tcp and nat causes problems with some sites
On Friday 21 July 2006 18:38, Walter Haidinger wrote: On Fri, 21 Jul 2006, Mike Frantzen wrote: Reassemble TCP does aggressive TCP PAWs checks on the TCP timestamps. It does the usual PAWs check to make sure a timestamp is not older than the last echoed value - which is in theory a wrapped sequence number. It also does its aggressive check to make sure the timestamp did not increase faster than the fastest clock which the RFC allows - an attacker can kill a connection by spoofing a higher timestamp. In order to prevent blind TCP spoofing PF's scrub will require TCP timestamps on all data packets if the first data packet had a timestamp. There are some transparent web caches out there that will allow the 3whs to complete (exchanging timestamp information) and then they will will hijack the TCP connection. Of course the developers of the hijacking device took some liberal shortcuts that make them look just like a blind hijacker instead of a MITM hijacker. If they hijack in the middle of the data stream (like after they see the HTTP request) and stop sending TCP Timestamps then PF's reassemble TCP will block further packets. How does this interfere with the NAT code? Reassemble tcp has _only_ problems from a machine behind the OpenBSD NAT router. On the router itself or if using a proxy running on it, there is no problem. Not sure whether this is relevant, but I had a similiar obscure 'bug' - some pages (from what I found most notably anything on *.free.fr) was timing out from behind the router, while the router itself was opening those pages without any issues. That behaviour disappeared when i commented out the first NAT rule I had in pf.conf: no nat on $ext_if from $int_if to $int_if Now everything works smoothly... Changing scrub one way or the other made no difference, most of the time I had in there: scrub in all So I'd think that it is actually an OpenBSD bug in the NAT and/or reassemble tcp code. How can this be verified? Regards, Walter -- viq
Re: scrub reassemble tcp and nat causes problems with some sites
Argh - It might help if I explain more. I have an OpenBSD 3.8 system running as a transparent packet filter (TPF). The OS X system is inside ($lanif). Apple's network - CIDR 17/8 is outside ($wanif). A Cisco PIX is doing NAT. IP's on the $wanif side that are inside the PIX are considered as DMZ. IP's on the $lanif side are considered LAN. WAN---PIX/NAT---DMZ---TPF---LAN---OS X Whenever I put a scrub rule with reassemble tcp on $wanif and/or $lanif I have trouble with some sites. (e.g. Apple's Software Update). setting debug to loud I get the messages I mention below. -Dan Daniel E. Hassler wrote: More info - I ran a test scenario. Here is a sample of the messages I get via syslog with set debug loud and scrub with reassemble tcp trying to run OS X's Software Update. Jul 19 19:42:37 obsd38 /bsd: pf_normalize_tcp_stateful: Did not receive expected RFC1323 timestamp Jul 19 19:42:37 obsd38 /bsd: TCP 192.168.1.14:65108 192.168.1.14:65108 17.250.248.95:80 [lo=4276925920 high=4276942304 win=65535 modulator=0 wscale=0] [lo=708430922 high=708496457 win=16384 modulator=0 wscale=0] 9:4 A -Dan Daniel E. Hassler wrote: Hi Walter, I've seen this behavior also. When I 'set debug loud' I got more information recorded via syslog. Some stuff about RFC1323 and bad-timestamp errors. Below is a section of a pf.conf file. It would be interesting to know if you get similar results with set debug loud when trying to access problem sites. # NORMALIZATION: reduce/resolve ambiguities. # scrub on $admif all random-id reassemble tcp #scrub on $lanif all random-id reassemble tcp #scrub on $wanif all random-id reassemble tcp # # Problem using reassemble tcp on $lanif and/or $wanif # Mac OS X software update fails. # bad-timestamp counter increments, RFC1323 errors in syslog with debug loud # All else works fine including other http on OS X. TBD: investigate further. # scrub on $lanif all random-id fragment reassemble scrub on $wanif all random-id fragment reassemble -Dan Walter Haidinger wrote: Hi! I'm running OpenBSD 3.9 GENERIC as a NAT router. If I add the reassemble tcp option to my scrub rule in pf.conf, I have trouble connecting to some sites, particulary ebay (ebay.de, ebay.at and ebay.com as well as e.g. kaufen.ebay.de) and some other few sites, from a machine behind the NAT router. Connects time out or have long delays if the site responds at all. If connecting directly from OpenBSD, using lynx or squid running on the router, there is no problem. If I omit reassemble tcp everything works fine, i.e. with: scrub all no-df fragment reassemble random-id I've never noticed the problem before because I was running the squid proxy on the router. Now I've moved it to a different machine which is NATted too. Please note that it is not a squid issue as timeouts occur regardless of proxy use if on a NATted machine. Unfortunately I cannot determine why only some sites have troubles and that's why I seeking advice here on howto further diagnose the problem. Any hints are appreciated! Regards, Walter -- _ _ _ __| | __ _ _ __ | |__ __ _ ___ ___| | ___ _ __ / _` |/ _` | '_ \ | '_ \ / _` / __/ __| |/ _ \ '__| | (_| | (_| | | | | | | | | (_| \__ \__ \ | __/ | \__,_|\__,_|_| |_| |_| |_|\__,_|___/___/_|\___|_| [EMAIL PROTECTED]
Re: scrub reassemble tcp and nat causes problems with some sites
You're going to have to turn off 'modulate tcp'. One of the TCP endpoints isn't following PAWs and stopped sending the TCP Timestamps or someone is trying to blind hijack the connection. More info - I ran a test scenario. Here is a sample of the messages I get via syslog with set debug loud and scrub with reassemble tcp trying to run OS X's Software Update. Jul 19 19:42:37 obsd38 /bsd: pf_normalize_tcp_stateful: Did not receive expected RFC1323 timestamp Jul 19 19:42:37 obsd38 /bsd: TCP 192.168.1.14:65108 192.168.1.14:65108 17.250.248.95:80 [lo=4276925920 high=4276942304 win=65535 modulator=0 wscale=0] [lo=708430922 high=708496457 win=16384 modulator=0 wscale=0] 9:4 A -Dan
Re: scrub reassemble tcp and nat causes problems with some sites
It's a stab in the dark but I would start with the assumption that some sites are using server load balancing and that reassemble tcp is breaking this somehow. Could be. Lets suspect poor load balancing because other big sites, which most likely do load balancing too, work. eBay is just the prime example where it does not... Then I'd try and prove that assumption by looking at the tcpdumps specifically for how reassemble tcp changes may be interfering. I'd have hoped that there is a less tedious solution... ;-) Get tcpdumps on both router interfaces with and without the reassemble tcp option. Do this for a similar file on both a working website and broken (ebay) website. On both router interfaces? Wouldn't the external if be enough? Tips on doing this: [well appreciated tips cut] Then load the comparable captures into Ethereal/Wireshark and stare at them until it makes sense :-) That's the tedious part! ;-) Thanks, Walter
Re: scrub reassemble tcp and nat causes problems with some sites
What is 'modulate tcp'? modulate state works fine. I get these errors only with scrub's reassemble tcp option I originally assumed it was an Apple problem since I only had trouble with the OS X Software Update feature. Going back to the beginning of this thread - Walter Haidinger appears to have a similar problem but not with Apple. I was hoping he could try 'set debug loud' in his pf.conf and check his /var/log/messages file after testing a problem site. If he sees messages similar to the one's I've seen maybe we both know a little more. -Dan Mike Frantzen wrote: You're going to have to turn off 'modulate tcp'. One of the TCP endpoints isn't following PAWs and stopped sending the TCP Timestamps or someone is trying to blind hijack the connection. More info - I ran a test scenario. Here is a sample of the messages I get via syslog with set debug loud and scrub with reassemble tcp trying to run OS X's Software Update. Jul 19 19:42:37 obsd38 /bsd: pf_normalize_tcp_stateful: Did not receive expected RFC1323 timestamp Jul 19 19:42:37 obsd38 /bsd: TCP 192.168.1.14:65108 192.168.1.14:65108 17.250.248.95:80 [lo=4276925920 high=4276942304 win=65535 modulator=0 wscale=0] [lo=708430922 high=708496457 win=16384 modulator=0 wscale=0] 9:4 A -Dan -- _ _ _ __| | __ _ _ __ | |__ __ _ ___ ___| | ___ _ __ / _` |/ _` | '_ \ | '_ \ / _` / __/ __| |/ _ \ '__| | (_| | (_| | | | | | | | | (_| \__ \__ \ | __/ | \__,_|\__,_|_| |_| |_| |_|\__,_|___/___/_|\___|_| [EMAIL PROTECTED]
scrub reassemble tcp and nat causes problems with some sites
Hi! I'm running OpenBSD 3.9 GENERIC as a NAT router. If I add the reassemble tcp option to my scrub rule in pf.conf, I have trouble connecting to some sites, particulary ebay (ebay.de, ebay.at and ebay.com as well as e.g. kaufen.ebay.de) and some other few sites, from a machine behind the NAT router. Connects time out or have long delays if the site responds at all. If connecting directly from OpenBSD, using lynx or squid running on the router, there is no problem. If I omit reassemble tcp everything works fine, i.e. with: scrub all no-df fragment reassemble random-id I've never noticed the problem before because I was running the squid proxy on the router. Now I've moved it to a different machine which is NATted too. Please note that it is not a squid issue as timeouts occur regardless of proxy use if on a NATted machine. Unfortunately I cannot determine why only some sites have troubles and that's why I seeking advice here on howto further diagnose the problem. Any hints are appreciated! Regards, Walter
Re: scrub reassemble tcp and nat causes problems with some sites
Walter Haidinger([EMAIL PROTECTED]) on 2006.07.19 12:28:52 +: Hi! I'm running OpenBSD 3.9 GENERIC as a NAT router. If I add the reassemble tcp option to my scrub rule in pf.conf, I have trouble connecting to some sites, particulary ebay (ebay.de, ebay.at and ebay.com as well as e.g. kaufen.ebay.de) and some other few sites, from a machine behind the NAT router. Connects time out or have long delays if the site responds at all. If connecting directly from OpenBSD, using lynx or squid running on the router, there is no problem. This sounds like a MTU problem. Either those sites are blocking ICMP-frag-needed messages or you are. - set the correct MTU - check pf.conf for scrub max-mss [...] - google - why do you use no-df? /B. [demime 1.01d removed an attachment of type application/pgp-signature]
Re: scrub reassemble tcp and nat causes problems with some sites
On Wed, 19 Jul 2006, Sebastian Benoit wrote: This sounds like a MTU problem. Either those sites are blocking Unlikely. I have cable, not a PPTP/PPPoE link. Therefore, no packet encapsulation. I'm aware of the MTU issue with ADSL. ICMP-frag-needed messages or you are. I think I am. _Only_ reassemble tcp breaks things, but why? - set the correct MTU - check pf.conf for scrub max-mss [...] No changes necessary, IMHO. - google Have done this, of course. Turned up e.g.: http://www.benzedrine.cx/pf/msg07352.html http://monkey.org/openbsd/archive/bugs/0312/msg00059.html Similar problem but no solution. - why do you use no-df? Because of the NFS issue mentionied in pf.conf(5) and the FAQ. May not be useful on the external interface, though. However, the problem persists even without no-df. Regards, Walter
Re: scrub reassemble tcp and nat causes problems with some sites
Unfortunately I cannot determine why only some sites have troubles and that's why I seeking advice here on howto further diagnose the problem. Any hints are appreciated! It's a stab in the dark but I would start with the assumption that some sites are using server load balancing and that reassemble tcp is breaking this somehow. Then I'd try and prove that assumption by looking at the tcpdumps specifically for how reassemble tcp changes may be interfering. Get tcpdumps on both router interfaces with and without the reassemble tcp option. Do this for a similar file on both a working website and broken (ebay) website. Tips on doing this: - be careful not to filter too much, you might miss an important icmp reply from an interim router - make sure tcpdump's snaplen is big enough to get all the headers - including http - try and replicate the issue with small html files so the packet captures aren't too busy - ensure that each capture sees the tcp handshake and FIN Then load the comparable captures into Ethereal/Wireshark and stare at them until it makes sense :-) Steve
Re: scrub reassemble tcp and nat causes problems with some sites
Hi Walter, I've seen this behavior also. When I 'set debug loud' I got more information recorded via syslog. Some stuff about RFC1323 and bad-timestamp errors. Below is a section of a pf.conf file. It would be interesting to know if you get similar results with set debug loud when trying to access problem sites. # NORMALIZATION: reduce/resolve ambiguities. # scrub on $admif all random-id reassemble tcp #scrub on $lanif all random-id reassemble tcp #scrub on $wanif all random-id reassemble tcp # # Problem using reassemble tcp on $lanif and/or $wanif # Mac OS X software update fails. # bad-timestamp counter increments, RFC1323 errors in syslog with debug loud # All else works fine including other http on OS X. TBD: investigate further. # scrub on $lanif all random-id fragment reassemble scrub on $wanif all random-id fragment reassemble -Dan Walter Haidinger wrote: Hi! I'm running OpenBSD 3.9 GENERIC as a NAT router. If I add the reassemble tcp option to my scrub rule in pf.conf, I have trouble connecting to some sites, particulary ebay (ebay.de, ebay.at and ebay.com as well as e.g. kaufen.ebay.de) and some other few sites, from a machine behind the NAT router. Connects time out or have long delays if the site responds at all. If connecting directly from OpenBSD, using lynx or squid running on the router, there is no problem. If I omit reassemble tcp everything works fine, i.e. with: scrub all no-df fragment reassemble random-id I've never noticed the problem before because I was running the squid proxy on the router. Now I've moved it to a different machine which is NATted too. Please note that it is not a squid issue as timeouts occur regardless of proxy use if on a NATted machine. Unfortunately I cannot determine why only some sites have troubles and that's why I seeking advice here on howto further diagnose the problem. Any hints are appreciated! Regards, Walter -- _ _ _ __| | __ _ _ __ | |__ __ _ ___ ___| | ___ _ __ / _` |/ _` | '_ \ | '_ \ / _` / __/ __| |/ _ \ '__| | (_| | (_| | | | | | | | | (_| \__ \__ \ | __/ | \__,_|\__,_|_| |_| |_| |_|\__,_|___/___/_|\___|_| [EMAIL PROTECTED]
Re: scrub reassemble tcp and nat causes problems with some sites
More info - I ran a test scenario. Here is a sample of the messages I get via syslog with set debug loud and scrub with reassemble tcp trying to run OS X's Software Update. Jul 19 19:42:37 obsd38 /bsd: pf_normalize_tcp_stateful: Did not receive expected RFC1323 timestamp Jul 19 19:42:37 obsd38 /bsd: TCP 192.168.1.14:65108 192.168.1.14:65108 17.250.248.95:80 [lo=4276925920 high=4276942304 win=65535 modulator=0 wscale=0] [lo=708430922 high=708496457 win=16384 modulator=0 wscale=0] 9:4 A -Dan Daniel E. Hassler wrote: Hi Walter, I've seen this behavior also. When I 'set debug loud' I got more information recorded via syslog. Some stuff about RFC1323 and bad-timestamp errors. Below is a section of a pf.conf file. It would be interesting to know if you get similar results with set debug loud when trying to access problem sites. # NORMALIZATION: reduce/resolve ambiguities. # scrub on $admif all random-id reassemble tcp #scrub on $lanif all random-id reassemble tcp #scrub on $wanif all random-id reassemble tcp # # Problem using reassemble tcp on $lanif and/or $wanif # Mac OS X software update fails. # bad-timestamp counter increments, RFC1323 errors in syslog with debug loud # All else works fine including other http on OS X. TBD: investigate further. # scrub on $lanif all random-id fragment reassemble scrub on $wanif all random-id fragment reassemble -Dan Walter Haidinger wrote: Hi! I'm running OpenBSD 3.9 GENERIC as a NAT router. If I add the reassemble tcp option to my scrub rule in pf.conf, I have trouble connecting to some sites, particulary ebay (ebay.de, ebay.at and ebay.com as well as e.g. kaufen.ebay.de) and some other few sites, from a machine behind the NAT router. Connects time out or have long delays if the site responds at all. If connecting directly from OpenBSD, using lynx or squid running on the router, there is no problem. If I omit reassemble tcp everything works fine, i.e. with: scrub all no-df fragment reassemble random-id I've never noticed the problem before because I was running the squid proxy on the router. Now I've moved it to a different machine which is NATted too. Please note that it is not a squid issue as timeouts occur regardless of proxy use if on a NATted machine. Unfortunately I cannot determine why only some sites have troubles and that's why I seeking advice here on howto further diagnose the problem. Any hints are appreciated! Regards, Walter -- _ _ _ __| | __ _ _ __ | |__ __ _ ___ ___| | ___ _ __ / _` |/ _` | '_ \ | '_ \ / _` / __/ __| |/ _ \ '__| | (_| | (_| | | | | | | | | (_| \__ \__ \ | __/ | \__,_|\__,_|_| |_| |_| |_|\__,_|___/___/_|\___|_| [EMAIL PROTECTED]
