Re: Having PF enabled breaks up rsync (and scp) over ssh connections
Hi, there was a stupid little typo in the pf rules I sent: the ssh port is obvioulys 22, not 20. That was just a typo in the e-mail, not on the server. -j. -- +358-404-177133 (WhatsApp) jyri.hov...@turvamies.fi
Having PF enabled breaks up rsync (and scp) over ssh connections
Hello everyone!
Now here's a mysterious one -- I've been working on this for weeks and still
have no clue what's causing it.
I have couple of virtual servers, running *the* latest 64-bit snapshots.
They're backed up using rsync over ssh. For a long time (months / years)
everything was working fine, but recently (maybe a couple of months ago) I
noticed that the rsync transfers keep getting cancelled.
I used to have a rather fancy /etc/pf.conf, with anti brute force stuff for
ssh, but even after disabling all that, the issue still persists.
As long os pf is enabled, rsync over ssh suddenly breaks down -- sometimes
before even transferring anything, sometimes after transferring couple of
megabytes, sometimes after tens or even hundreds of megabytes.
On client side, the following error is shown:
"client_loop: send disconnect: Broken pipe
rsync: connection unexpectedly closed (15936376 bytes received so far)
[receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226)
[receiver=3.1.3]
rsync: connection unexpectedly closed (1076081 bytes received so far)
[generator]
rsync error: unexplained error (code 255) at io.c(226) [generator=3.1.3]"
On server side, the sshd logs the error message: "process_output:
ssh_packet_write_poll: Connection from user TheUserName xxx.xxx.xxx.xxx port
n: Permission denied"
I tried running the ssh daemon in debug mode, but even then the best I could
get was the above permission denied error -- no further details.
As soon as I disable pf entirely, the problem goes away.
Unlike the rsync over ssh sessions, my normal ssh console sessions stay awake
without any problems.
Looking at the traffic with tcpdump reveals that the server is sending TCP
resets when the connection breaks down.
Just to make sure it's not a memory issue, the user that's used to do the
backups is in the staff login class.
Also, to make sure the problem is not in rsync, I tried using scp with the
exact same results.
Here's my simplified /etc/pf.conf:
set reassemble yes
set block-policy drop
set loginterface egress
block drop log all label default_deny
block return out log proto {tcp udp} user _pbuild
match in all scrub (no-df random-id max-mss 1440)
block in quick log from urpf-failed label uRPF_check_failed
pass in quick log (to pflog0) on egress proto tcp \
from any port > 1023 \
to (egress) port 20 \
user root \
flags S/SA keep state
Any ideas on how to debug this further?
Yours,
Jyri
--
+358-404-177133 (WhatsApp)
jyri.hov...@turvamies.fi
Re: Web documentation available offline by default?
On Fri, 28 Feb 2020 at 18:14, Luke A. Call wrote: > > Another option I found helpful once is to use wget to download the > FAQs' content to a local copy (unless that puts too much load on the server), > then have a simple local shell alias to view it with links or w3m. > (At the time, it was a quick way for me, to preserve the content > in case I wanted it while offline, or if things like X weren't working.) > There are probably pros & cons of doing that, vs. CVS -- maybe making a > CVS copy is actually cleaner & simpler for this, and for updating it. > > I can fish out my old wget line for that, if it is of interest and not > considered harmful. It's also a pity the the faq are not available in a single html or pdf format. This would be handy for those who, like me, are studying for the BSD Specialist certification. Having a single document makes it easier to search for a specific command. There's a one-page text file on the ftp server but this is quite old (it doesn't even mention doas). -- Ottavio Caruso
Re: Catastrophic
Yeah like Stuart said I need to reproduce the crash and get inside the machine when it’s in that state. To be continued. Best On Fri, Feb 28, 2020 at 7:42 PM Avon Robertson wrote: > On Sat, Feb 29, 2020 at 12:57:07AM +1000, Stuart Longland wrote: > > On 28/2/20 11:32 pm, Justin Noor wrote: > > > Thanks for offering to help and sorry for the delay - I got dragged > into a > > > work emergency. I finally managed to SCP my dmesg to a remote machine. > > > > Heh, no problems, these things happen. > > > > > As a refresher I have a 6.6 current machine that crashes when X is > running, > > > and almost instantly when Firefox is running - it runs fine without X. > The > > > machine becomes totally frozen - I have to perform a forced shutdown to > > > exit this state. The issue appears to be graphics related and is > > > inconsistent - sometimes it crashes immediately, other times it does > not. > > > > Sometimes it might be the way a particular graphics toolkit "tickles" > > the video hardware too. For instance FVWM uses libxcb for drawing > > graphics which means you're likely to be just working with 2D primitives. > > > > Then Firefox with its GTK+ back-end fires off a few RENDER extension > > requests to the X server and whoopsie! Down she goes! > > > > > There are indeed some "unknown product" messages related to my PCI > graphics > > > card in my dmesg, but I haven't been able to decipher them yet. Those > > > usually mean the device is not supported, but it is, and I'm sure I > have > > > the correct driver (amdgpu0). Previously I had no issues for months, > which > > > is why I suspected hardware failure. Admittedly I've been lucky with > > > graphics cards over the years, and don't know much about PCI. > > > > No issues for months running a previous version of OpenBSD or the same > > you're running now? > > > > One suggestion I made too was to maybe try setting up a serial console > > link… turns out the motherboard makers know how to tease: > > > > > com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo > > > com0: probed fifo depth: 0 bytes > > > > That says there is a RS-232 port somewhere… so I had a look at the > handbook: > > > https://dlcdnets.asus.com/pub/ASUS/mb/SocketAM4/ROG_STRIX_B450-I_GAMING/E14337_ROG_STRIX_B450-I_GAMING_UM_PRINT.pdf > > > > They didn't wire it up to a pin header, which is annoying. > > > > On the video front, I did see this: > > > initializing kernel modesetting (POLARIS11 0x1002:0x67EF 0x1002:0x0B04 > > > 0xE5). > > > amdgpu_irq_add_domain: stub > > > amdgpu_device_resize_fb_bar: stub > > > amdgpu: [powerplay] Failed to retrieve minimum clocks. > > > amdgpu0: 1360x768, 32bpp > > > wsdisplay0 at amdgpu0 mux 1: console (std, vt100 emulation), using > wskbd0 > > > wskbd1: connecting to wsdisplay0 > > > wsdisplay0: screen 1-5 added (std, vt100 emulation) > > > > The "stub" messages make me wonder if we're hitting some > > not-yet-implemented features. That "failed to retrieve minimum clocks" > > has been seen on Linux as well, and there it was related to PCI prefetch > > register programming. > > > > The machine you've got isn't much different to what I have at work > > actually: Rysen 7 1700 (so previous generation), and a RX550 video card > > (POLARIS12, maybe slightly newer?)… the machine is fitted with a RS-232 > > serial port so I might try a little experiment with a USB stick and see > > if I can install OpenBSD 6.6 to USB storage and try to reproduce the > crash. > > -- > > Stuart Longland (aka Redhatter, VK4MSL) > > > > I haven't lost my mind... > > ...it's backed up on a tape somewhere. > > > > Hello Justin and Stuart, > > I hope the following may be of help in solving the cause of the crash. > > I have experienced a similar type of crash when using X on this machine > for approximately the last 6 weeks. Prior to this, X had been running on > this machine without apparent problems for 12 plus months. > > The only browser installed on this machine is lynx(1). My crashes have > been random with no recognised culprit at the time of the crash, which > usually occurred within 10 minutes of invoking startx(1). > > fvwm(1) is the only window manager installed on this machine. All my > crashes have required the machine to be powered off to regain control. > > This machine's graphics card was identified by it's vendor as a: > Sapphire Nitro+ RX580 8G GDDR5 Graphics Card 2X HDMI + 2X Display+DVI > Port. > This machine is connected to it's monitor using a Display Port cable. > > This machine has worked and is working without problems from a console, > with and without tmux(1). If multiple consoles are run at the same time > however, when exit(3) is invoked from one of them the time taken to > exit is sometimes longer than 10 seconds. This seems odd to me. > > Please find below the contents of this machine's /var/run/dmesg.boot. > > OpenBSD 6.6-current (GENERIC.MP) #0: Sun Feb 23 00:07:16 MST 2020 >
Re: Catastrophic
Awesome - thank you for your time and for the valuable information. That’s hilarious about the serial port. I’ll try plugging into a switch, reproducing the crash, and SSHing into it. I still haven’t tried the syslogd tip you mentioned either. It’s time for me to start learning more about X. Will be in touch. Regards On Fri, Feb 28, 2020 at 6:57 AM Stuart Longland wrote: > On 28/2/20 11:32 pm, Justin Noor wrote: > > Thanks for offering to help and sorry for the delay - I got dragged into > a > > work emergency. I finally managed to SCP my dmesg to a remote machine. > > Heh, no problems, these things happen. > > > As a refresher I have a 6.6 current machine that crashes when X is > running, > > and almost instantly when Firefox is running - it runs fine without X. > The > > machine becomes totally frozen - I have to perform a forced shutdown to > > exit this state. The issue appears to be graphics related and is > > inconsistent - sometimes it crashes immediately, other times it does not. > > Sometimes it might be the way a particular graphics toolkit "tickles" > the video hardware too. For instance FVWM uses libxcb for drawing > graphics which means you're likely to be just working with 2D primitives. > > Then Firefox with its GTK+ back-end fires off a few RENDER extension > requests to the X server and whoopsie! Down she goes! > > > There are indeed some "unknown product" messages related to my PCI > graphics > > card in my dmesg, but I haven't been able to decipher them yet. Those > > usually mean the device is not supported, but it is, and I'm sure I have > > the correct driver (amdgpu0). Previously I had no issues for months, > which > > is why I suspected hardware failure. Admittedly I've been lucky with > > graphics cards over the years, and don't know much about PCI. > > No issues for months running a previous version of OpenBSD or the same > you're running now? > > One suggestion I made too was to maybe try setting up a serial console > link… turns out the motherboard makers know how to tease: > > > com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo > > com0: probed fifo depth: 0 bytes > > That says there is a RS-232 port somewhere… so I had a look at the > handbook: > > https://dlcdnets.asus.com/pub/ASUS/mb/SocketAM4/ROG_STRIX_B450-I_GAMING/E14337_ROG_STRIX_B450-I_GAMING_UM_PRINT.pdf > > They didn't wire it up to a pin header, which is annoying. > > On the video front, I did see this: > > initializing kernel modesetting (POLARIS11 0x1002:0x67EF 0x1002:0x0B04 > > 0xE5). > > amdgpu_irq_add_domain: stub > > amdgpu_device_resize_fb_bar: stub > > amdgpu: [powerplay] Failed to retrieve minimum clocks. > > amdgpu0: 1360x768, 32bpp > > wsdisplay0 at amdgpu0 mux 1: console (std, vt100 emulation), using wskbd0 > > wskbd1: connecting to wsdisplay0 > > wsdisplay0: screen 1-5 added (std, vt100 emulation) > > The "stub" messages make me wonder if we're hitting some > not-yet-implemented features. That "failed to retrieve minimum clocks" > has been seen on Linux as well, and there it was related to PCI prefetch > register programming. > > The machine you've got isn't much different to what I have at work > actually: Rysen 7 1700 (so previous generation), and a RX550 video card > (POLARIS12, maybe slightly newer?)… the machine is fitted with a RS-232 > serial port so I might try a little experiment with a USB stick and see > if I can install OpenBSD 6.6 to USB storage and try to reproduce the crash. > -- > Stuart Longland (aka Redhatter, VK4MSL) > > I haven't lost my mind... > ...it's backed up on a tape somewhere. >
Re: What TERM fixes Emacs?
On 2020-02-27, Stuart Longland wrote: > On 26/2/20 9:46 pm, Marc Espie wrote: >> (these days, new OS versions will all use the same termcap source, so you're >> probably safe on anything released over the past 5 years) People with a need to connect to older OS (from the text console rather than X) probably have enough grey in their beards to cope anyway ;) The problem is more that some OS default to installing a cut-down set of terminfo files rather than the whole thing. (e.g. for rhel7 you need to install the ncurses-term package - no idea about rhel8 - looks like debian is ok). > Just thinking… a suitable work-around for such OSes would be to set the > following in ~/.ssh/config: > > Host my.old.host > SetEnv TERM=vt220 Or just connect from inside tmux, which uses "screen" variants by default that are widely supported. (And of course this does not affect X). > Note I haven't tested this to see if it works, just read `man > ssh_config`. Obviously this doesn't help with other use cases like > `telnet`. For those, one could do `TERM=vt100 telnet foo.example.com 1234`. > > If newer OSes already understand TERM=pccon, perhaps in the medium term > it might be worth reviewing this default setting. I think keeping it at vt220 now causes more problems that would be fixed by moving to pccon, than would be caused by changing to pccon.
