Re: Having PF enabled breaks up rsync (and scp) over ssh connections
Are you using PF rules with "overload"? If so, try removing that. On 2020-03-06, Jyri Hovila [Turvamies.fi] wrote: > Hi! > >> Thanks, and sorry I missed in your previous mail that things were working >> OK before. In which case it doesn't sound like a configuration problem >> which I wasn't sure about. > > Thank you for helping in the debugging, and no need for apologies! > >> Do you have any logs that might help tie it down to a shorter timeframe? >> (kernel messages at boot ae written to /var/log/messages, so the snapshot >> date will show up there if the logs haven't rotated away yet). > > I'll take a look if I can find anything related there. > >> I think it would be good to move this to bugs@ as it won't get the attention >> it needs on misc - can you collate things into one mail and include a dmesg >> please? > > I think you're right. > > Before I do that, though, I'll upgrade to the very latest snapshot. > > Thanks again! > > -j. > -- > +63-905-3119715 (Mobile) > +358-404-177133 (WhatsApp) > jyri.hov...@turvamies.fi > >
Re: Having PF enabled breaks up rsync (and scp) over ssh connections
Hi! > Thanks, and sorry I missed in your previous mail that things were working > OK before. In which case it doesn't sound like a configuration problem > which I wasn't sure about. Thank you for helping in the debugging, and no need for apologies! > Do you have any logs that might help tie it down to a shorter timeframe? > (kernel messages at boot ae written to /var/log/messages, so the snapshot > date will show up there if the logs haven't rotated away yet). I'll take a look if I can find anything related there. > I think it would be good to move this to bugs@ as it won't get the attention > it needs on misc - can you collate things into one mail and include a dmesg > please? I think you're right. Before I do that, though, I'll upgrade to the very latest snapshot. Thanks again! -j. -- +63-905-3119715 (Mobile) +358-404-177133 (WhatsApp) jyri.hov...@turvamies.fi
Re: Having PF enabled breaks up rsync (and scp) over ssh connections
On 2020/03/06 18:53, Jyri Hovila [Turvamies.fi] wrote: > Hi! > > > Look at pfctl -ss -v. Do you have "wscale" values printed for most > > TCP connections? > > There's wscale for all the active connections, at least at the moment. > > > If not then you are likely creating state on intermediate > > rather than ACK packets (window scaling values are not present in most > > packets, only the initial handshake). If that is the case, make sure > > you don't have anything passed by the implicit default rule which is > > equivalent to "pass all flags any no state", > > I don't pass anything without a state. > > > I like to make sure I don't > > hit this by starting my ruleset with "block log". > > This has always been the default approach for me, since day 1. > > Note that I've used more or less the same setup, based on and refined from > several "howto do it properly with pf" articles, for literally decades. It's > the first time ever I'm experiencing the issue. Since I'm following CURRENT, > I'm leaning towards the conclusion that rather than anything being wrong with > my ruleset, something has changed in the kernel / pf. Disclaimer: can't be > sure, of course. > > > Note the paragraph starting "Where more than one firewall might actively > > handle packets" in pfsync(4) if this might apply to you. > > There's nothing like that going on here; it's a traditional single-host setup. > > Actually, I'm experiencing the same issue on two hosts -- both of which were > functioning fine until about a month ago this problem popped up. > > > You can also try bumping up the PF debug level (pfctl -x; the default > > is "err"), be careful doing this on a busy system (take it up one notch > > at a time and make sure it doesn't create insane amounts of logging, be > > ready to knock back to a previous level if needed). Be extra careful > > with this if you have serial console. > > I'll try this. > > -j. > Thanks, and sorry I missed in your previous mail that things were working OK before. In which case it doesn't sound like a configuration problem which I wasn't sure about. Do you have any logs that might help tie it down to a shorter timeframe? (kernel messages at boot ae written to /var/log/messages, so the snapshot date will show up there if the logs haven't rotated away yet). I think it would be good to move this to bugs@ as it won't get the attention it needs on misc - can you collate things into one mail and include a dmesg please?
Re: Having PF enabled breaks up rsync (and scp) over ssh connections
Hi! > Look at pfctl -ss -v. Do you have "wscale" values printed for most > TCP connections? There's wscale for all the active connections, at least at the moment. > If not then you are likely creating state on intermediate > rather than ACK packets (window scaling values are not present in most > packets, only the initial handshake). If that is the case, make sure > you don't have anything passed by the implicit default rule which is > equivalent to "pass all flags any no state", I don't pass anything without a state. > I like to make sure I don't > hit this by starting my ruleset with "block log". This has always been the default approach for me, since day 1. Note that I've used more or less the same setup, based on and refined from several "howto do it properly with pf" articles, for literally decades. It's the first time ever I'm experiencing the issue. Since I'm following CURRENT, I'm leaning towards the conclusion that rather than anything being wrong with my ruleset, something has changed in the kernel / pf. Disclaimer: can't be sure, of course. > Note the paragraph starting "Where more than one firewall might actively > handle packets" in pfsync(4) if this might apply to you. There's nothing like that going on here; it's a traditional single-host setup. Actually, I'm experiencing the same issue on two hosts -- both of which were functioning fine until about a month ago this problem popped up. > You can also try bumping up the PF debug level (pfctl -x; the default > is "err"), be careful doing this on a busy system (take it up one notch > at a time and make sure it doesn't create insane amounts of logging, be > ready to knock back to a previous level if needed). Be extra careful > with this if you have serial console. I'll try this. -j.
Re: Having PF enabled breaks up rsync (and scp) over ssh connections
On 2020-03-03, Chris Cappuccio wrote: > Jyri Hovila [Turvamies.fi] [jyri.hov...@turvamies.fi] wrote: >> Hello everyone! >> >> Now here's a mysterious one -- I've been working on this for weeks and still >> have no clue what's causing it. >> >> "client_loop: send disconnect: Broken pipe >> >> As soon as I disable pf entirely, the problem goes away. >> >> Any ideas on how to debug this further? >> > > Figure out which exact part of your pf config is causing this. Try disabling > everything line-by-line. > > Look at pfctl -ss -v. Do you have "wscale" values printed for most TCP connections? If not then you are likely creating state on intermediate rather than ACK packets (window scaling values are not present in most packets, only the initial handshake). If that is the case, make sure you don't have anything passed by the implicit default rule which is equivalent to "pass all flags any no state", I like to make sure I don't hit this by starting my ruleset with "block log". Note the paragraph starting "Where more than one firewall might actively handle packets" in pfsync(4) if this might apply to you. You can also try bumping up the PF debug level (pfctl -x; the default is "err"), be careful doing this on a busy system (take it up one notch at a time and make sure it doesn't create insane amounts of logging, be ready to knock back to a previous level if needed). Be extra careful with this if you have serial console.
Re: Having PF enabled breaks up rsync (and scp) over ssh connections
Jyri Hovila [Turvamies.fi] [jyri.hov...@turvamies.fi] wrote: > Hello everyone! > > Now here's a mysterious one -- I've been working on this for weeks and still > have no clue what's causing it. > > "client_loop: send disconnect: Broken pipe > > As soon as I disable pf entirely, the problem goes away. > > Any ideas on how to debug this further? > Figure out which exact part of your pf config is causing this. Try disabling everything line-by-line.
Re: Having PF enabled breaks up rsync (and scp) over ssh connections
Hi, there was a stupid little typo in the pf rules I sent: the ssh port is obvioulys 22, not 20. That was just a typo in the e-mail, not on the server. -j. -- +358-404-177133 (WhatsApp) jyri.hov...@turvamies.fi
Having PF enabled breaks up rsync (and scp) over ssh connections
Hello everyone!
Now here's a mysterious one -- I've been working on this for weeks and still
have no clue what's causing it.
I have couple of virtual servers, running *the* latest 64-bit snapshots.
They're backed up using rsync over ssh. For a long time (months / years)
everything was working fine, but recently (maybe a couple of months ago) I
noticed that the rsync transfers keep getting cancelled.
I used to have a rather fancy /etc/pf.conf, with anti brute force stuff for
ssh, but even after disabling all that, the issue still persists.
As long os pf is enabled, rsync over ssh suddenly breaks down -- sometimes
before even transferring anything, sometimes after transferring couple of
megabytes, sometimes after tens or even hundreds of megabytes.
On client side, the following error is shown:
"client_loop: send disconnect: Broken pipe
rsync: connection unexpectedly closed (15936376 bytes received so far)
[receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226)
[receiver=3.1.3]
rsync: connection unexpectedly closed (1076081 bytes received so far)
[generator]
rsync error: unexplained error (code 255) at io.c(226) [generator=3.1.3]"
On server side, the sshd logs the error message: "process_output:
ssh_packet_write_poll: Connection from user TheUserName xxx.xxx.xxx.xxx port
n: Permission denied"
I tried running the ssh daemon in debug mode, but even then the best I could
get was the above permission denied error -- no further details.
As soon as I disable pf entirely, the problem goes away.
Unlike the rsync over ssh sessions, my normal ssh console sessions stay awake
without any problems.
Looking at the traffic with tcpdump reveals that the server is sending TCP
resets when the connection breaks down.
Just to make sure it's not a memory issue, the user that's used to do the
backups is in the staff login class.
Also, to make sure the problem is not in rsync, I tried using scp with the
exact same results.
Here's my simplified /etc/pf.conf:
set reassemble yes
set block-policy drop
set loginterface egress
block drop log all label default_deny
block return out log proto {tcp udp} user _pbuild
match in all scrub (no-df random-id max-mss 1440)
block in quick log from urpf-failed label uRPF_check_failed
pass in quick log (to pflog0) on egress proto tcp \
from any port > 1023 \
to (egress) port 20 \
user root \
flags S/SA keep state
Any ideas on how to debug this further?
Yours,
Jyri
--
+358-404-177133 (WhatsApp)
jyri.hov...@turvamies.fi
