Re: Help with server not accepting new connections but is still accessible through ONE existing open ssh-session

2017-02-09 Thread lilit-aibolit 

On 02/01/2017 03:41 PM, Erling Westenvik wrote:

I have an OpenBSD 5.9 server at a colocation. It stopped accepting new
connections (ping, ssh, http, whatever) yesterday night but fortunately
I had one ssh session open from my workstation from which I can still
access it.


Did you think about creation of second sshd instance
on other port and start it in debug mode?

Re: Help with server not accepting new connections but is still accessible through ONE existing open ssh-session

2017-02-01 Thread Erling Westenvik 
On Wed, Feb 01, 2017 at 05:09:43PM +0200, Lars Noodén wrote:
> On 02/01/2017 05:06 PM, Erling Westenvik wrote:
> > On Wed, Feb 01, 2017 at 03:58:51PM +0100, Manuel Giraud wrote:
> >> Erling Westenvik  writes:
> >>
> >>> However, I got inspired and when I disabled pf (pfctl -d) I got full
> >>> contact! (But -- when I turned pf back on (pfctl -e) I lost the one
> >>> connection I had... Now I have to wait 48 minutes for the server to
> >>> reboot. Not much more to do now except for crossing my fingers...)
> >>
> >> Err, yes but won't pf be enabled at boot time? Hopefully, some of your
> >> pf tables will be reset.
> >
> > True. But before I turned pf off and back on, I couldn't be sure what
> > was causing the problem. If it was an external problem I would've been
> > better off with the one active existing ssh connection.
> >
>
> I hope it reboots ok.  If you end up with a similar situation again you
> might set up 2 or more at jobs that build reverse tunnels from port 22
> to an outside machine.  That way you can still connect back via a tunnel
> if the main SSH session drops.  That won't solve the problem but might
> buy you more time to investigate.
>
> /Lars

Thanks. Good advice and I actually HAD such a tunnel to a remote
machine earlier to day but took it down due to experimenting, but
without remembering to reactivate it. Won't forget that again!

--
Erling Westenvik

Re: Help with server not accepting new connections but is still accessible through ONE existing open ssh-session

2017-02-01 Thread Erling Westenvik 
On Wed, Feb 01, 2017 at 03:58:51PM +0100, Manuel Giraud wrote:
> Erling Westenvik  writes:
>
> > However, I got inspired and when I disabled pf (pfctl -d) I got full
> > contact! (But -- when I turned pf back on (pfctl -e) I lost the one
> > connection I had... Now I have to wait 48 minutes for the server to
> > reboot. Not much more to do now except for crossing my fingers...)
>
> Err, yes but won't pf be enabled at boot time? Hopefully, some of your
> pf tables will be reset.

True. But before I turned pf off and back on, I couldn't be sure what
was causing the problem. If it was an external problem I would've been
better off with the one active existing ssh connection.

--
Erling Westenvik

Re: Help with server not accepting new connections but is still accessible through ONE existing open ssh-session

2017-02-01 Thread Manuel Giraud 
Erling Westenvik  writes:

> However, I got inspired and when I disabled pf (pfctl -d) I got full
> contact! (But -- when I turned pf back on (pfctl -e) I lost the one
> connection I had... Now I have to wait 48 minutes for the server to
> reboot. Not much more to do now except for crossing my fingers...)

Err, yes but won't pf be enabled at boot time? Hopefully, some of your
pf tables will be reset.
-- 
Manuel Giraud

Re: Help with server not accepting new connections but is still accessible through ONE existing open ssh-session

2017-02-01 Thread Erling Westenvik 
On Wed, Feb 01, 2017 at 04:26:15PM +0200, lilit-aibolit wrote:
> On 02/01/2017 03:41 PM, Erling Westenvik wrote:
> > I have an OpenBSD 5.9 server at a colocation. It stopped accepting new
> > connections (ping, ssh, http, whatever) yesterday night but fortunately
> > I had one ssh session open from my workstation from which I can still
> > access it.
> >
> Did you think about creation of second sshd instance
> on other port and start it in debug mode?

Thank you for answering.

No, it didn't occur to me since I could not reach the machine by any
service or port. Tried your suggestion immediately without success.

However, I got inspired and when I disabled pf (pfctl -d) I got full
contact! (But -- when I turned pf back on (pfctl -e) I lost the one
connection I had... Now I have to wait 48 minutes for the server to
reboot. Not much more to do now except for crossing my fingers...)

Thanks anyway! :-)

--
Erling Westenvik

Help with server not accepting new connections but is still accessible through ONE existing open ssh-session

2017-02-01 Thread Erling Westenvik 
I have an OpenBSD 5.9 server at a colocation. It stopped accepting new
connections (ping, ssh, http, whatever) yesterday night but fortunately
I had one ssh session open from my workstation from which I can still
access it. Funny thing is that the server has full access OUT to the
internet. I can open web pages through lynx, ssh to everywhere, and so
on. It just won't accept any new connections IN.

The colocation provider claim that nothing has changed at their side.
(Gateway, firewall, DNS, etc.) Since the location for the server is not
easily accessible, and in a worst case scenario wouldn't be accessible
for many days or even a week, I'd rather try to find and solve the
problem before having to resort to a reboot. (In case the machine
doesn't come up again, leaving me without the one ssh session that is
alive as for now.)

Pflog/tcpdump shows absolutely NO activity, neither in nor out. That is
strange IMO and I'm suspecting that some states in pf may be the
problem. I'm tempted to do a pfctl -F all, but that may also kill the
only ssh session I have open. (I'm resetting shutdown -r +60 every now
and then so that the server will at least do a reboot if the ssh
connection should fail.)

Any ideas as to where to begin?

--
Erling Westenvik


$ uptime
 2:39PM  up 253 days,  2:15, 1 user, load averages: 0.27, 0.28, 0.22


$ dmesg
OpenBSD 5.9 (GENERIC.MP) #1888: Fri Feb 26 01:20:19 MST 2016
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 8505982976 (8111MB)
avail mem = 8243998720 (7862MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.3 @ 0xfbae0 (60 entries)
bios0: vendor American Megatrends Inc. version "080011" date 06/30/2006
bios0: Supermicro H8DSP-8
acpi0 at bios0: rev 0
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP APIC OEMB SRAT
acpi0: wakeup devices P1P2(S4) USB0(S1) USB1(S1) USB2(S1) PS2K(S4) PS2M(S4) 
BR14(S4) BR1E(S4) BR28(S4) BR3C(S4) SLPB(S4)
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD Opteron(tm) Processor 250, 2394.33 MHz
cpu0: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW,LAHF
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB 64b/line 
16-way L2 cache
cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu0: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu0: AMD erratum 89 present, BIOS upgrade may be required
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 199MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: AMD Opteron(tm) Processor 250, 2394.00 MHz
cpu1: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW,LAHF
cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB 64b/line 
16-way L2 cache
cpu1: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu1: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu1: AMD erratum 89 present, BIOS upgrade may be required
ioapic0 at mainbus0: apid 2 pa 0xfec0, version 11, 16 pins
ioapic1 at mainbus0: apid 3 pa 0xfec01000, version 11, 16 pins
ioapic2 at mainbus0: apid 4 pa 0xfec02000, version 11, 16 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (P0P1)
acpiprt2 at acpi0: bus 2 (P1P2)
acpiprt3 at acpi0: bus 3 (BR14)
acpiprt4 at acpi0: bus 4 (BR1E)
acpiprt5 at acpi0: bus 5 (BR28)
acpiprt6 at acpi0: bus 6 (BR32)
acpiprt7 at acpi0: bus 7 (BR3C)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpibtn0 at acpi0: PWRB
acpibtn1 at acpi0: SLPB
pci0 at mainbus0 bus 0
ppb0 at pci0 dev 1 function 0 "ServerWorks HT-1000 PCI" rev 0x00
pci1 at ppb0 bus 1
ppb1 at pci1 dev 13 function 0 "ServerWorks HT-1000 PCIX" rev 0xb2
pci2 at ppb1 bus 2
pciide0 at pci1 dev 14 function 0 "ServerWorks HT-1000 SATA" rev 0x00: DMA
pciide0: using apic 2 int 11 for native-PCI interrupt
pciide0: port 0: 1.5Gb/s
wd0 at pciide0 channel 0 drive 0: 
wd0: 16-sector PIO, LBA48, 476940MB, 976773168 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 6
pciide0: port 1: 1.5Gb/s
wd1 at pciide0 channel 1 drive 0: 
wd1: 16-sector PIO, LBA48, 476940MB, 976773168 sectors
wd1(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 6
pciide0: port 2: 1.5Gb/s
wd2 at pciide0 channel 2 drive 0: 
wd2: 16-sector PIO, LBA48, 476940MB, 976773168 sectors
wd2(pciide0:2:0): using PIO mode 4, Ultra-DMA mode 6
pciide0: port 3: 1.5Gb/s
wd3 at pciide0 channel 3 drive 0: 
wd3: 16-sector PIO, LBA48, 476940MB, 976773168 sectors
wd3(pciide0:3:0): using PIO mode 4, Ultra-DMA mode 6
pciide1 at pci1 dev 14 function 1 "ServerWorks HT-1000 SATA" rev 0x00
piixpm0 at pci0 dev 2 function 0 "ServerWorks HT-1000" rev 0x00: polling
iic0 at piixpm0
iic0: addr 0x1b 0f=18