Re: Having PF enabled breaks up rsync (and scp) over ssh connections
Are you using PF rules with "overload"? If so, try removing that. On 2020-03-06, Jyri Hovila [Turvamies.fi] wrote: > Hi! > >> Thanks, and sorry I missed in your previous mail that things were working >> OK before. In which case it doesn't sound like a configuration problem >> which I wasn't sure about. > > Thank you for helping in the debugging, and no need for apologies! > >> Do you have any logs that might help tie it down to a shorter timeframe? >> (kernel messages at boot ae written to /var/log/messages, so the snapshot >> date will show up there if the logs haven't rotated away yet). > > I'll take a look if I can find anything related there. > >> I think it would be good to move this to bugs@ as it won't get the attention >> it needs on misc - can you collate things into one mail and include a dmesg >> please? > > I think you're right. > > Before I do that, though, I'll upgrade to the very latest snapshot. > > Thanks again! > > -j. > -- > +63-905-3119715 (Mobile) > +358-404-177133 (WhatsApp) > jyri.hov...@turvamies.fi > >
Re: Having PF enabled breaks up rsync (and scp) over ssh connections
Hi! > Thanks, and sorry I missed in your previous mail that things were working > OK before. In which case it doesn't sound like a configuration problem > which I wasn't sure about. Thank you for helping in the debugging, and no need for apologies! > Do you have any logs that might help tie it down to a shorter timeframe? > (kernel messages at boot ae written to /var/log/messages, so the snapshot > date will show up there if the logs haven't rotated away yet). I'll take a look if I can find anything related there. > I think it would be good to move this to bugs@ as it won't get the attention > it needs on misc - can you collate things into one mail and include a dmesg > please? I think you're right. Before I do that, though, I'll upgrade to the very latest snapshot. Thanks again! -j. -- +63-905-3119715 (Mobile) +358-404-177133 (WhatsApp) jyri.hov...@turvamies.fi
Re: Having PF enabled breaks up rsync (and scp) over ssh connections
On 2020/03/06 18:53, Jyri Hovila [Turvamies.fi] wrote: > Hi! > > > Look at pfctl -ss -v. Do you have "wscale" values printed for most > > TCP connections? > > There's wscale for all the active connections, at least at the moment. > > > If not then you are likely creating state on intermediate > > rather than ACK packets (window scaling values are not present in most > > packets, only the initial handshake). If that is the case, make sure > > you don't have anything passed by the implicit default rule which is > > equivalent to "pass all flags any no state", > > I don't pass anything without a state. > > > I like to make sure I don't > > hit this by starting my ruleset with "block log". > > This has always been the default approach for me, since day 1. > > Note that I've used more or less the same setup, based on and refined from > several "howto do it properly with pf" articles, for literally decades. It's > the first time ever I'm experiencing the issue. Since I'm following CURRENT, > I'm leaning towards the conclusion that rather than anything being wrong with > my ruleset, something has changed in the kernel / pf. Disclaimer: can't be > sure, of course. > > > Note the paragraph starting "Where more than one firewall might actively > > handle packets" in pfsync(4) if this might apply to you. > > There's nothing like that going on here; it's a traditional single-host setup. > > Actually, I'm experiencing the same issue on two hosts -- both of which were > functioning fine until about a month ago this problem popped up. > > > You can also try bumping up the PF debug level (pfctl -x; the default > > is "err"), be careful doing this on a busy system (take it up one notch > > at a time and make sure it doesn't create insane amounts of logging, be > > ready to knock back to a previous level if needed). Be extra careful > > with this if you have serial console. > > I'll try this. > > -j. > Thanks, and sorry I missed in your previous mail that things were working OK before. In which case it doesn't sound like a configuration problem which I wasn't sure about. Do you have any logs that might help tie it down to a shorter timeframe? (kernel messages at boot ae written to /var/log/messages, so the snapshot date will show up there if the logs haven't rotated away yet). I think it would be good to move this to bugs@ as it won't get the attention it needs on misc - can you collate things into one mail and include a dmesg please?
Re: Having PF enabled breaks up rsync (and scp) over ssh connections
Hi! > Look at pfctl -ss -v. Do you have "wscale" values printed for most > TCP connections? There's wscale for all the active connections, at least at the moment. > If not then you are likely creating state on intermediate > rather than ACK packets (window scaling values are not present in most > packets, only the initial handshake). If that is the case, make sure > you don't have anything passed by the implicit default rule which is > equivalent to "pass all flags any no state", I don't pass anything without a state. > I like to make sure I don't > hit this by starting my ruleset with "block log". This has always been the default approach for me, since day 1. Note that I've used more or less the same setup, based on and refined from several "howto do it properly with pf" articles, for literally decades. It's the first time ever I'm experiencing the issue. Since I'm following CURRENT, I'm leaning towards the conclusion that rather than anything being wrong with my ruleset, something has changed in the kernel / pf. Disclaimer: can't be sure, of course. > Note the paragraph starting "Where more than one firewall might actively > handle packets" in pfsync(4) if this might apply to you. There's nothing like that going on here; it's a traditional single-host setup. Actually, I'm experiencing the same issue on two hosts -- both of which were functioning fine until about a month ago this problem popped up. > You can also try bumping up the PF debug level (pfctl -x; the default > is "err"), be careful doing this on a busy system (take it up one notch > at a time and make sure it doesn't create insane amounts of logging, be > ready to knock back to a previous level if needed). Be extra careful > with this if you have serial console. I'll try this. -j.
Re: Having PF enabled breaks up rsync (and scp) over ssh connections
On 2020-03-03, Chris Cappuccio wrote: > Jyri Hovila [Turvamies.fi] [jyri.hov...@turvamies.fi] wrote: >> Hello everyone! >> >> Now here's a mysterious one -- I've been working on this for weeks and still >> have no clue what's causing it. >> >> "client_loop: send disconnect: Broken pipe >> >> As soon as I disable pf entirely, the problem goes away. >> >> Any ideas on how to debug this further? >> > > Figure out which exact part of your pf config is causing this. Try disabling > everything line-by-line. > > Look at pfctl -ss -v. Do you have "wscale" values printed for most TCP connections? If not then you are likely creating state on intermediate rather than ACK packets (window scaling values are not present in most packets, only the initial handshake). If that is the case, make sure you don't have anything passed by the implicit default rule which is equivalent to "pass all flags any no state", I like to make sure I don't hit this by starting my ruleset with "block log". Note the paragraph starting "Where more than one firewall might actively handle packets" in pfsync(4) if this might apply to you. You can also try bumping up the PF debug level (pfctl -x; the default is "err"), be careful doing this on a busy system (take it up one notch at a time and make sure it doesn't create insane amounts of logging, be ready to knock back to a previous level if needed). Be extra careful with this if you have serial console.
Re: Having PF enabled breaks up rsync (and scp) over ssh connections
Jyri Hovila [Turvamies.fi] [jyri.hov...@turvamies.fi] wrote: > Hello everyone! > > Now here's a mysterious one -- I've been working on this for weeks and still > have no clue what's causing it. > > "client_loop: send disconnect: Broken pipe > > As soon as I disable pf entirely, the problem goes away. > > Any ideas on how to debug this further? > Figure out which exact part of your pf config is causing this. Try disabling everything line-by-line.
Re: Having PF enabled breaks up rsync (and scp) over ssh connections
Hi, there was a stupid little typo in the pf rules I sent: the ssh port is obvioulys 22, not 20. That was just a typo in the e-mail, not on the server. -j. -- +358-404-177133 (WhatsApp) jyri.hov...@turvamies.fi
