SysKonnect sk driver high CPU interrupt rate after 3.8 upgrade

[Paul B. Henson]

We have been running a pair of firewalls under 3.3 for quite some time, and
just upgraded them to 3.8. Before the upgrade, they would typically run at
about 35-40% interrupts. After the upgrade, they are running at 90-95%
interrupt utilization.

It looks like a lot of changes have been made to the sk driver over the
past couple of years, apparently for the worse for our particular cards :(.

The cards in particular are:

skc0 at pci6 dev 1 function 0 Schneider  Koch 984x GE rev 0x12: irq 11
skc0: SysKonnect GEnesis (0x0)
sk0 at skc0 port A: address 00:00:5a:9a:7f:d4
xmphy0 at sk0 phy 0: XaQti Corp. XMAC II Gigabit PHY, rev. 2


Any suggestions on reducing the interrupt load to the previous levels?

Thanks...


-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  [EMAIL PROTECTED]
California State Polytechnic University  |  Pomona CA 91768

Sun x4100 amd64 virtual console -- uhidev0: bad input length 8 != 0

[Paul B. Henson]

I just installed OpenBSD 4.0-stable on a Sun x4100 server. The server has
an IPMI virtual console accessed remotely via a Java application which
pretends to be a USB keyboard/mouse.

While I had no problems with the virtual console during the installation
process, once the actual operating system was installed, every keypress on
the virtual console results in the message uhidev0: bad input length 8 != 0
being displayed.

Other than the message being displayed, the keyboard seems to be working
fine. I removed the following code from uhidev.c, which removed the
message logging. As far as I can tell, the keyboard works fine under the
virtual console, although I haven't tested the mouse emulation. I guess
there's just something weird about the USB emulation. No functional
problems, but if whoever maintains the USB driver is interested in figuring
it out, I could test patches...


#ifdef DIAGNOSTIC
  if (scd-sc_in_rep_size != cc)
printf(%s: bad input length %d != %d\n,USBDEVNAME(sc-sc_dev),
   scd-sc_in_rep_size, cc);
#endif




-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  [EMAIL PROTECTED]
California State Polytechnic University  |  Pomona CA 91768

Sun x4100 amd64 dies with NMI under heavy network load

[Paul B. Henson]

 0xfe9ff000, version 11, 4 pins
ioapic4 at mainbus0 apid 8 pa 0xfe9fe000, version 11, 4 pins
pci0 at mainbus0 bus 0: configuration mode 1
ppb0 at pci0 dev 1 function 0 AMD 8131 PCIX rev 0x13
pci1 at ppb0 bus 1
em0 at pci1 dev 1 function 0 Intel PRO/1000MT (82546EB) rev 0x03: apic 5
int 2 (irq 10), address 00:14:4f:20:98:90
em1 at pci1 dev 1 function 1 Intel PRO/1000MT (82546EB) rev 0x03: apic 5
int 3 (irq 11), address 00:14:4f:20:98:91
em2 at pci1 dev 2 function 0 Intel PRO/1000MT (82546EB) rev 0x03: apic 5
int 0 (irq 11), address 00:14:4f:20:98:96
em3 at pci1 dev 2 function 1 Intel PRO/1000MT (82546EB) rev 0x03: apic 5
int 1 (irq 9), address 00:14:4f:20:98:97
aapic0 at pci0 dev 1 function 1 AMD 8131 PCIX IOAPIC rev 0x01
ppb1 at pci0 dev 2 function 0 AMD 8131 PCIX rev 0x13
pci2 at ppb1 bus 2
em4 at pci2 dev 1 function 0 Intel PRO/1000MF (82545GM) rev 0x04: apic 6
int 0 (irq 11), address 00:04:23:ce:03:78
mpi0 at pci2 dev 3 function 0 Symbios Logic SAS1064 rev 0x02: apic 6 int
0 (irq 11)
scsibus0 at mpi0: 63 targets
sd0 at scsibus0 targ 0 lun 0: LSILOGIC, Logical Volume, 3000 SCSI2
0/direct fixed
sd0: 34332MB, 34332 cyl, 16 head, 128 sec, 512 bytes/sec, 70311936 sec
total
aapic1 at pci0 dev 2 function 1 AMD 8131 PCIX IOAPIC rev 0x01
ppb2 at pci0 dev 6 function 0 AMD 8111 PCI-PCI rev 0x07
pci3 at ppb2 bus 3
ohci0 at pci3 dev 0 function 0 AMD 8111 USB rev 0x0b: apic 4 int 19 (irq
11), version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
uhub0 at usb0
uhub0: AMD OHCI root hub, rev 1.00/1.00, addr 1
uhub0: 3 ports with 3 removable, self powered
ohci1 at pci3 dev 0 function 1 AMD 8111 USB rev 0x0b: apic 4 int 19 (irq
11), version 1.0, legacy support
usb1 at ohci1: USB revision 1.0
uhub1 at usb1
uhub1: AMD OHCI root hub, rev 1.00/1.00, addr 1
uhub1: 3 ports with 3 removable, self powered
vga1 at pci3 dev 3 function 0 ATI Rage XL rev 0x27
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 7 function 0 AMD AMD8111 LPC rev 0x05
pciide0 at pci0 dev 7 function 1 AMD 8111 IDE rev 0x03: DMA, channel 0
configured to compatibility, channel 1 configured to compatibility
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 ignored (disabled)
amdiic0 at pci0 dev 7 function 2 AMD 8111 SMBus rev 0x02: SCI
iic0 at amdiic0: disabled to avoid ipmi0 interactions
amdpm0 at pci0 dev 7 function 3 AMD 8111 Power rev 0x05: rng active
iic1 at amdpm0: disabled to avoid ipmi0 interactions
pchb0 at pci0 dev 24 function 0 AMD AMD64 HyperTransport rev 0x00
pci4 at pchb0 bus 4
ppb3 at pci4 dev 1 function 0 AMD 8131 PCIX rev 0x13
pci5 at ppb3 bus 5
aapic2 at pci4 dev 1 function 1 AMD 8131 PCIX IOAPIC rev 0x01
ppb4 at pci4 dev 2 function 0 AMD 8131 PCIX rev 0x13
pci6 at ppb4 bus 6
em5 at pci6 dev 1 function 0 Intel PRO/1000MF (82546GB) rev 0x03: apic 8
int 1 (irq 9), address 00:04:23:c2:e0:ac
em6 at pci6 dev 1 function 1 Intel PRO/1000MF (82546GB) rev 0x03: apic 8
int 2 (irq 10), address 00:04:23:c2:e0:ad
aapic3 at pci4 dev 2 function 1 AMD 8131 PCIX IOAPIC rev 0x01
pchb1 at pci0 dev 24 function 1 AMD AMD64 Address Map rev 0x00
pchb2 at pci0 dev 24 function 2 AMD AMD64 DRAM Cfg rev 0x00
pchb3 at pci0 dev 24 function 3 AMD AMD64 Misc Cfg rev 0x00
pchb4 at pci0 dev 25 function 0 AMD AMD64 HyperTransport rev 0x00
pchb5 at pci0 dev 25 function 1 AMD AMD64 Address Map rev 0x00
pchb6 at pci0 dev 25 function 2 AMD AMD64 DRAM Cfg rev 0x00
pchb7 at pci0 dev 25 function 3 AMD AMD64 Misc Cfg rev 0x00
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: PC speaker
spkr0 at pcppi0
uhidev0 at uhub0 port 3 configuration 1 interface 0umass0 at uhub1 port 1
configuration 1 interface 0
uhidev0: American Megatrends Inc. Virtual Keyboard and Mouse, rev
1.10/1.00, addr 2, iclass 3/1

umass0: American Megatrends Inc. Virtual Cdrom Device, rev 1.10/1.00, addr
2
umass0: using ATAPI over Bulk-Only
scsibus1 at umass0: 2 targets
ukbd0 at uhidev0: 8 modifier keys, 6 key codes
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub0 port 3 configuration 1 interface 1
uhidev1: American Megatrends Inc. Virtual Keyboard and Mouse, rev
1.10/1.00, addr 2, iclass 3/1
ums0 at uhidev1
ums0: X report 0x0002 not supported
cd0 at scsibus1 targ 1 lun 0: AMI, Virtual CDROM, 1.00 SCSI0 5/cdrom
removable
umass1 at uhub1 port 2 configuration 1 interface 0
umass1: American Megatrends Inc. Virtual Floppy Device, rev 1.10/1.00, addr
3
umass1: using UFI over CBI with CCI
scsibus2 at umass1: 2 targets
sd1 at scsibus2 targ 1 lun 0: AMI, Virtual Floppy, 1.00 SCSI0 0/direct
removable
sd1: drive offline
dkcsum: sd0 matches BIOS drive 0x80
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02


-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating

Re: Sun x4100 amd64 dies with NMI under heavy network load

[Paul B. Henson]

On Thu, 16 Nov 2006, Srebrenko Sehic wrote:

  When booting, RTC BIOS diagnostic error 2 is displayed, I'm not sure if
  that's relevant.

 You might want to investigate that. Not sure, but I don't remember
 seeing that error on the X4200 boxes I had tested. BIOS update might
 be relevant. Perhaps it's also caused by bad hardware.

I just installed on an identical system, which gives the same error during
boot :(. They both have the latest bios installed.

 I have not seen any stability problems with my X4200 deployments. They
 are not running as network firewalls, but as application level
 proxies, so the error you are seeing could be due to higher pps count.
 Unlike you, I didn't put anything non-stock in the box. 4 built-in
 NICs where enough for my purposes.

The system seems to run stable until I put load on the multiport fiber
adapter. I recompiled the entire operating system with no issues.

 USB layer doesn't work in ddb so you'll need the serial working to
 get useful debug data.

Did you get serial working on your x4200 boxes? I tried not configuring the
OpenBSD serial console and using BIOS redirection, that didn't work. I also
tried disabling BIOS redirection and configuring explicit OpenBSD serial
console support, but still nothing showed up when I connected to the remote
management console interface. The only thing I haven't tried is physically
connecting to the local serial port on the hardware itself.


-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  [EMAIL PROTECTED]
California State Polytechnic University  |  Pomona CA 91768

root/boot on softraid in 5.0

[Paul B. Henson]

I've been running openbsd 4.6 for a couple years now with root on
softraid, booting off a CF card with a kernel compiled to hardcode
root/swap on sd0.

I read about official support for root on softraid:

http://www.undeadly.org/cgi?action=articlesid=20111002154251

and got the impression it would just work, particularly the part about
eliminates the need for a custom kernel.

However, I just did a test install on a vm with two ide hard drives (wd0
and wd1) configured into a softraid mirror (sd0), and when booting the kernel
from wd0a it tries to find the root on wd0a as well, and panics.

I was able to get it to boot by either providing the -a option to boot
and specifying sd0a as the root, or by compiling a custom kernel with
sd0a hardcoded as I did in 4.6.

Am I missing something? Based on the web post, I expected the kernel
loaded from wd0a to figure out root was on sd0a and boot successfully.

Looking at the underlying commit:

http://article.gmane.org/gmane.os.openbsd.cvs/108176

It's talking about comparing the rootduid to the softraid volume. I'm
not clear where this is coming from, the fstab in sd0a uses duid's, but
I don't see how the booting kernel would know about that yet.

Anyway, just to clarify my understanding, is it expected in 5.0 to be
able to boot softraid root without a custom kernel or using -a? If so,
what am I doing wrong?

Thanks...


-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  hen...@csupomona.edu
California State Polytechnic University  |  Pomona CA 91768

Re: root/boot on softraid in 5.0

[Paul B. Henson]

On Wed, Dec 21, 2011 at 08:08:08PM -0800, Josh Grosse wrote:

 Woops.  I misread your post.  The commits were September 19, which is
 -current, beyond 5.0-release.
 
 You must either migrate to -current, or await 5.1-release.

Ah, ok, thanks for the clarification. The installboot piece that lets
you install bootblocks on softraid is in 5.0, so when that part worked I
assumed it all was. The dates (commits in Sept, your post in Oct,
5.0 release in Nov) also led me to misbelieve it was in 5.0. But looking
at the changelogs I see the bits that store boot info in softraid
metadata and dynamically figure out the root happened after the 5.0
freeze.

Something to look forward to in 5.1 :). Thanks again...


-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  hen...@csupomona.edu
California State Polytechnic University  |  Pomona CA 91768

Re: acpitz0: _AL0[0] _PR0 failed on intel atom mb

[Paul B. Henson]

On Thu, 21 Jan 2010, Brynet wrote:

 Several BIOS updates appear available for your motherboard as well
 perhaps one of them will solve the problem

Thanks for the pointer, I had the latest when I looked but I guess 04 came
out since then. In fact, when I went to download it today, it had already
jumped to 05. I spent *way* too much time updating the bios :(, it's an
embedded box with no floppy (just an internal HD and a CF in a CF-IDE
adapter). I lost count of the contortions of boot images and boot methods
(USB cdrom, various CF images) I went through but finally managed to get
the latest bios on, with no change in behavior sigh.

 Nothing in the ReadMe.txt files indicates any ACPI related fixes.. but
 A04 does seem to add some kind of support for a Winbond sensor.

Looks like that Winbond part number is a flash chip.

 You should probably try a -CURRENT snapshot, before reporting problems
 you see in 4.6.

After updating the bios I booted the latest 4.6-current kernel, which still
had the problem, so I submitted a bug.

Thanks much for the help...


-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  hen...@csupomona.edu
California State Polytechnic University  |  Pomona CA 91768

ospf broken on trunk interfaces?

[Paul B. Henson]

I'm trying to setup ospf on a trunk interface. I've had it configured
and working fine on a regular interface for quite some time, and now am
trying to add another neighbor on a trunk interface, and it just shows
the interface as down:

# ospfctl show i
Interface   AddressState  HelloTimer Linkstate  Uptimenc ac
trunk0  10.128.0.9/30  DOWN   -  active 00:00:00   0 0
lo1 10.128.0.4/32  LOOP   -  unknown17w5d04h   0 0
re0 10.128.0.1/30  BCKUP  00:00:05   active 17w5d08h   1 1

The trunk is definitely up:

# ifconfig trunk0
trunk0: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500
lladdr 00:30:18:a8:7c:cc
priority: 0
trunk: trunkproto lacp
trunk id: [(8000,00:30:18:a8:7c:cc,4094,,),
 (8000,f0:25:72:53:82:00,0001,,)]
trunkport re1 active,collecting,distributing
groups: trunk
media: Ethernet autoselect
status: active
inet 10.128.0.9 netmask 0xfffc broadcast 10.128.0.11
inet6 fe80::230:18ff:fea8:7ccc%trunk0 prefixlen 64 scopeid 0x12

I currently only have one physical port in the trunk (planning to add a second
later once it's all working):

# ifconfig re1
re1: flags=8b43UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST mtu 
1500
lladdr 00:30:18:a8:7c:cc
priority: 0
trunk: trunkdev trunk0
media: Ethernet autoselect (1000baseT full-duplex)
status: active
inet6 fe80::230:18ff:feac:b83a%re1 prefixlen 64 scopeid 0x2

Traffic is definitely passing, I can ping the other side (which is a cisco 
layer3
switch):

# ping 10.128.0.10
PING 10.128.0.10 (10.128.0.10): 56 data bytes
64 bytes from 10.128.0.10: icmp_seq=0 ttl=255 time=4.427 ms

tcpdump on trunk0 shows ospf hello packets from the cisco gear:

22:26:09.862595 cisco-bart.pbhware.com  ospf-all.mcast.net: OSPFv2-hello  
44[80]: rtrid cisco.nms.pbhware.com backbone dr cisco-bart.pbhware.com [tos 
0xc0] [ttl 1]

I found this mailing list posting with exactly the same problem it would seem:

http://old.nabble.com/trunk-and-ospf-on-openbsd-4.8-td31833059.html

But there were no responses. I'm running OpenBSD 5.0, and in case I'm doing
something stupid here's the ospf config:

router-id 10.128.0.4
redistribute default
redistribute connected

area 0.0.0.0 {

interface lo1:10.128.0.4 { passive }

interface re0 {
auth-type crypt
auth-md 1 X
auth-md-keyid 1
}
interface trunk0 {
auth-type crypt
auth-md 1 X
auth-md-keyid 1
}
}


Any suggestions? Is ospf not supported on trunk interfaces as surmised by the
other mailing list posting?

Thanks much for any assistance...

Re: ospf broken on trunk interfaces?

[Paul B. Henson]

On Sun, Jun 03, 2012 at 11:04:53AM +, Stuart Henderson wrote:

 Did you create the trunk interface *before* or *after* starting ospfd?
 I have seen ospfd get the wrong state on interfaces created after startup,
 iirc sometimes ifconfig down+up helps, sometimes you need to restart
 ospfd.

The trunk was created after ospfd was started. I didn't even think of
restarting ospfd, but that did the trick, once it was restarted it found
the new neighbor on the other side of the trunk and all the routes
propogated between them (the only caveat was I didn't think about the
routes disappearing when I killed ospfd, so I lost my connection to the
box 8-/, but fortunately the console wasn't too inconvenient to get to).

Thanks much for your timely and accurate suggestion, hopefully google
will find it for the next person who might have this issue :).

Re: ospf broken on trunk interfaces?

[Paul B. Henson]

On Sun, Jun 03, 2012 at 05:12:19PM +0200, Claudio Jeker wrote:

 Is this after a reload of the config or does this also happen when you
 restart ospfd?

It was after a config reload, after following Stuart's suggestion to
restart ospfd everything's working great :). Maybe it would be worth a
note in the ospfctl man page that sometimes a reload isn't sufficient,
and ospfd might need to be completely restarted for an interface that's
created after it's already running?

Thanks...

Intel Atom S1260 (SuperServer 5017A-EF)

[Paul B. Henson]

I'm looking at a supermicro SuperServer 5017A-EF for openbsd purposes,
it's got an Intel atom S1260 SoC, Marvell 88SE9230 SATA, and i350AM2 dual
gig interfaces.

It looks like i350 support shipped in 5.2, and I'm pretty sure the
Marvell chip is AHCI compliant, so I'd think that would be ok, but I'm
leery about the SoC, I can't find any references to openbsd running on
this specific chip or any atom based SoC for that matter and I'd hate to
buy a box that didn't run openbsd well :(.

Any feedback on this particular server, this atom SoC in specific, or
even a general opinion on how well this might work out much appreciated :).

Thanks much...

Re: Intel Atom S1260 (SuperServer 5017A-EF)

[Paul B. Henson]

On Fri, Nov 15, 2013 at 11:25:50PM +0100, Sebastian Benoit wrote:

 Don't buy this one (yet). The Marvell 88SE9230 SATA does not work.
 i know cause i have one ;-)

Arg, disappointing, but I'm glad I thought to check before buying :). Do
you know if anybody's working on it? So much for standard AHCI sigh,
does it not find it, or find it but crap out? Do all the other
components work ok? I could temporarily stick a PCI SATA card in it to
get by until the onboard SATA is supported if all the other pieces are
happy. Does anybody have any suggestions for a good/cheap 2 port SATA
PCI card that supports openbsd?

 The earlier 5017A-* machines are ok.

Hmm, the only other 5017A model I see doesn't have IPMI.

Thanks for the help...

Re: Intel Atom S1260 (SuperServer 5017A-EF)

[Paul B. Henson]

On Fri, Nov 15, 2013 at 11:25:50PM +0100, Sebastian Benoit wrote:

 Don't buy this one (yet). The Marvell 88SE9230 SATA does not work.
 i know cause i have one ;-)

Hmm, looks like support was added in FreeBSD back in June 2012:

http://lists.freebsd.org/pipermail/svn-src-stable-9/2012-June/002131.html

so hopefully it wouldn't be to hard for somebody with the right skill
set (unfortunately not me when it comes to low level drivers sigh) to
tune it up for openbsd. Looking at the backstory behind that commit:

http://forums.freebsd.org/showthread.php?t=32563

evidentally marvell doesn't follow the AHCI spec very well and the
freebsd driver has workarounds for various quirks. Stupid marvell :(,
too bad supermicro didn't use a better sata chip.

Poking through the freebsd code, it looks like it has a workaround for
Marvell controllers do not wait for readyness which appears to be
adding in an extra delay when the controller is reset, and Some weird
controllers do not return signature in FIS receive area. Read it from
PxSIG register., which copies some results from a different location
overwriting what was copied in from the standard location. Other than
that, I don't see any other kludges, the rest is just the standard ahci
stuff. I see the openbsd ahci driver is completely different than the
freebsd one, so dunno how easily such workarounds could be implemented.

Re: Intel Atom S1260 (SuperServer 5017A-EF)

[Paul B. Henson]

On Sat, Nov 16, 2013 at 11:34:15AM +0100, Sebastian Benoit wrote:

 sorry, i mispoke, i meant 5015A-* and they dont have a dedicated ipmi port.

Oh, yah, I've actually got one of those, it's been working great. I was
actually planning on replacing it with this newer one, which supports
more memory and has more power, and reallocate it to another task.

 anyway, dmesg attached, if someone cares. i'm not going to do anything more
 with it.

 cpu0: apic clock running at 99MHz
 cpu at mainbus0: not configured
 cpu at mainbus0: not configured
 cpu at mainbus0: not configured

 ahci0: failed to stop port, cannot softreset

Hmm, not very promising, it didn't even initialize all four cores. The
ahci error is one of the things the freebsd driver works around, the
crappy marvell chipset breaks spec on the reset function.

Lots of unknowns and unconfigured in that dmesg :(, guess I need to
find another option. Least I found out before I bought it, thanks much
for the heads up.

Re: Intel Atom S1260 (SuperServer 5017A-EF)

[Paul B. Henson]

On Sat, Nov 16, 2013 at 12:27:08PM +0100, Carsten Larsen wrote:

 Maybe just buy the previous model 5015A-*? I have been running one of 
 those for some years now and it works like a charm. From their website I 
 see it has reached End-of-Life though.

I've actually got one of those, as you say, I've been very happy with
it. I was looking for a newer model with more power and a separate IPMI
port. Guess I've got to keep looking...

Re: Intel Atom S1260 (SuperServer 5017A-EF)

[Paul B. Henson]

On Fri, Nov 15, 2013 at 08:42:50PM -0800, Chris Cappuccio wrote:

 It's very old. This patch did not make it into the driver and I have
 no idea if those chips work through some other change, or not. Likely
 not. These older chips must be really buggy pieces of shit if you have
 to disable NCQ.

Bleh. I can definitely see the openbsd philosophy leaning towards not
supporting crap ;). The two workarounds in freebsd for this newer marvell
sata chipset don't seem quite as egregious, but I'm not really a low level
driver guy...

Re: Intel Atom S1260 (SuperServer 5017A-EF)

[Paul B. Henson]

On Sat, Nov 16, 2013 at 12:15:19PM -0800, Paul B. Henson wrote:

  sorry, i mispoke, i meant 5015A-* and they dont have a dedicated ipmi port.
 
 Oh, yah, I've actually got one of those, it's been working great. I was
 actually planning on replacing it with this newer one, which supports
 more memory and has more power, and reallocate it to another task.

I forgot to mention, but the newer one also supports ECC memory, which
is a plus.

low-power/small form factor server (supermicro X9SCL-F w Core i3-3220T)

[Paul B. Henson]

I was recently looking for a low-power small form factor box and was
initially thinking of the supermicro SuperServer 5017A-EF, which seemed a
good fit. Unfortunately, the fairly new atom SoC in that box isn't currently
supported, nor is the crappy not-quite-AHCI Marvell sata controller. So,
I'm thinking of putting something together from parts instead.

I'm looking at the supermicro X9SCL-F motherboard which has an Intel C202
PCH chipset and 2 gigabit interfaces (Intel 82579LM and 82574L), combined
with a Core i3-3220T, stuffed in a 510T-203B chassis.

I see from the em man page and the list archives that those two Intel
ethernet chipsets seem reasonably well supported. I couldn't find any
specific mention of the C202 chipset, but I believe the Intel AHCI SATA
interface is actually AHCI compliant, so trust it would work fine with the
standard ahci driver. The i3 processor has a 35w TDP versus the atom's 8.5w,
but actually working with openbsd is a bit more important than saving a few
watts :).

According to the Intel ARK this i3 processor should support ECC memory when
installed on a board with a server class chipset. I really appreciated the
heads up I got last week about the unsupported atom, that definitely saved
me from ordering a box I couldn't use 8-/, so if anybody sees any potential
issues with this combination for an openBSD server I'd appreciate hearing
about it :).

Thanks much.

Re: low-power/small form factor server (supermicro X9SCL-F w Core i3-3220T)

[Paul B. Henson]

 From: Bryan Vyhmeister [mailto:br...@bsdjournal.net]
 Sent: Tuesday, November 19, 2013 9:46 PM

 I have lots of X9SCL-F, X9SCL+-F, X9SCM-F, X9SCI-LN4, X9SCI-LN4F,
 X9SCM-iiF boards running OpenBSD in production. Both network interfaces
 work flawlessly.

Cool, thanks much for the info.

 Although I'm not using any of the low power chips since I've
 found that heat is really not an issue and the non T chips scale down

With the 200W power supply in the small form factor chassis, supermicro says
the max processor TDP supported by the motherboard is 45w. I guess if you
put one in that potentially uses greater power but never push it to do so it
would still work, but I assume bad things would happen if it ever
accidentally cranked and tried to suck more power than was available 8-/.

 G860, Core i3 2120, Core i3 3240, Xeon E3 1220, Xeon E3 1260L, and Xeon

The specifications for the motherboard on the supermicro site say processors
with integrated graphics are not recommended, and since so far I've been
unable to push them into clarifying why I was a little leery. The footnote
indicates it's coming from Intel regarding the C202 chipset. I've seen a
handful of reports of people using processors with integrated graphics with
this chipset, and then with your confirmation I feel better about ordering
it. Obviously the chipset doesn't support integrated graphics, so the
silicon in the CPU is going to waste, but I'm guessing the supermicro
documentation team read doesn't support integrated graphics in the chipset
documentation and translated that into you shouldn't use one as opposed to
if you do use one, you can't use the integrated graphics.

 If you don't need IPMI, you could save a few dollars and go with the non
 F versions of the boards. I have found that the IPMI Text Console
 never works right for anything I've tried including OpenBSD.

I've used the serial redirection on illumos and linux boxes without any
trouble. I rarely use the video redirection other than for potentially
initial bootstrapping and rare diagnostic issues. It's nice not ever having
to visit the box in person once it's racked :).

Thanks again for the feedback, it was very helpful.

Re: low-power/small form factor server (supermicro X9SCL-F w Core i3-3220T)

[Paul B. Henson]

 From: Stuart Henderson
 Sent: Wednesday, November 20, 2013 3:54 AM

 One thing to note, which may be irrelevant, but may be very important,
 is which CPUs support AES-NI - the LGA1155 Pentium/i3 don't.

Yeah, you've got to bump up to a much more expensive Xeon to get that :(.
Thanks for the heads up, but for this box the extra cost isn't worth it.

Re: low-power/small form factor server (supermicro X9SCL-F w Core i3-3220T)

[Paul B. Henson]

On Wed, Nov 20, 2013 at 12:35:35PM -0800, 'Bryan Vyhmeister' wrote:

 From looking at Supermicro's CSE-510-203B page, it says 65W TDP and
 every CPU I've mentioned below except for the Xeon E3 1220 (80W) and
 Xeon E3 1230v2 (69W) fall below this.

Hmm, I guess I was actually looking at the SuperServer 5017C-LF page:

http://www.supermicro.com/products/system/1U/5017/SYS-5017C-LF.cfm

It has the X9SCL-F motherboard, a similar chassis with a 200w power supply,
and indicates max tdp = 45w. I asked supermicro support about the
510T-203B chassis with the same motherboard, and they told me it only
supported up to 45w as well. Dunno, better safe than sorry, the T
version is about the same price as the regular.

Re: low-power/small form factor server (supermicro X9SCL-F w Core i3-3220T)

[Paul B. Henson]

 From: 'Bryan Vyhmeister' [mailto:br...@bsdjournal.net]
 Sent: Wednesday, November 20, 2013 1:51 PM

 Very interesting. There is some ambiguity in the specs. Looking at the
 SC510L-200B chassis which is what's included with the SYS-5017C-LF
 system you linked to, it also says 65W TDP.

Well, it can't hurt to use less power :). I ended up ordering all the parts,
hopefully they'll show up by mid next week so I can assemble them over the
long weekend.

I managed to escalate the integrated graphics question high enough to find
somebody who knew what they were talking about, he said, as you confirmed,
that they work fine with this motherboard other than that you cannot use the
integrated graphics, it is disabled. Pretty much what I thought, they should
clarify their footnote on the specification page.

 the hw.sensors values:

Out of curiosity, have you tried enabling the ipmi driver? I have an older
atom server with an X7SPA-HF motherboard (which is actually being replaced
by this one), and I found that the ipmi sensor provided more values than the
lm one. The box came as passively cooled, I ended up sticking in three fans
anyway. It still runs a bit hot, but within the acceptable range for that
processor:

hw.sensors.ipmi0.temp0=54.00 degC (System Temp), CRITICAL
hw.sensors.ipmi0.temp1=64.00 degC (CPU Temp), CRITICAL
hw.sensors.ipmi0.fan0=4840 RPM (CPU FAN), OK
hw.sensors.ipmi0.fan1=6135 RPM (SYS FAN), OK
hw.sensors.ipmi0.volt0=1.11 VDC (CPU Vcore), OK
hw.sensors.ipmi0.volt1=1.04 VDC (Vichcore), OK
hw.sensors.ipmi0.volt2=3.30 VDC (+3.3VCC), OK
hw.sensors.ipmi0.volt3=1.53 VDC (VDIMM), OK
hw.sensors.ipmi0.volt4=5.09 VDC (+5 V), OK
hw.sensors.ipmi0.volt5=12.30 VDC (+12 V), OK
hw.sensors.ipmi0.volt6=3.30 VDC (+3.3VSB), OK
hw.sensors.ipmi0.volt7=3.22 VDC (VBAT), OK
hw.sensors.ipmi0.indicator0=Off (Chassis Intru), OK
hw.sensors.ipmi0.indicator1=On (PS Status), OK

 I think the SC510T-200B/203B is the better choice for hard drives. I use

Yep. I like hot-swap, it would be a pain to have to disconnect and unrack
the unit to swap out a fixed internal drive.

Re: low-power/small form factor server (supermicro X9SCL-F w Core i3-3220T)

[Paul B. Henson]

On Wed, Nov 20, 2013 at 10:16:05PM -0500, Ted Unangst wrote:

 The ipmi driver is disabled by default because it does bad things on
 some systems. If you don't go out of your way to enable it, the not
 configured line is all you'll see.

That's what I was going to say, but you beat me to it ;). For mailing
list archive purposes, you need to run 'config -e' on your kernel binary
and 'enable ipmi', then reboot.

So far I haven't found anything bad it does on my system. When it's
enabled, there is a watchdog setting in sysctl on my box, but the last
time I tried to use it it always rebooted after the interval, it didn't
seem to be reseting the timeout correctly. My box hasn't ever frozen since
it's been deployed, so that lack hasn't been a problem. The only
difference I've noticed between ipmi/no ipmi is a few more sensors
displayed.

Re: Patch to remove adult content from spamd(8) man page

[Paul B. Henson]

On Fri, Nov 22, 2013 at 01:09:36PM -0600, J. Lewis Muir wrote:

 I don't see it that way.  Huckleberry Finn is a book, and I don't need
 to read it unless I want to.  The spamd(8) man page is a man page I need
 to read in order to understand how to use spamd.

Let me fix that for you:

The spamd(8) man page is a man page I don't need to read it unless I
want to use spamd, a choice I am making of my own free will, and if I
don't like it, I guess I could just go use some other software that
doesn't get my panties in a bunch.

Maybe you could try spam assassin instead? Unless, of course, you find
the metaphor of killing spam offensive...

Intel 82574L vs 82579LM

[Paul B. Henson]

I've got a box with one Intel 82574L based ethernet port and one 82579LM
based ethernet port. One will be hooked up to a 50Mbps wan link, the
other trunked at gigabit speed to a cisco switch (both routing the wan
link, routing some internal vlans, and providing some services).

Both the 82574L and 82579LM work fine, but I was wondering if one was
better than the other as far as performance or cpu utilization, so I
could make that one the internal link and the other the wan link,
otherwise I guess I'll just hook them up whichever way makes the cables
shorter ;).

Thanks...

IPMI SOL serial console wedges

[Paul B. Henson]

I've got a supermicro X9SCL-F board with ipmi support, and I'm trying to
use it for the serial console. It shows up as a third com port. After
booting the latest install cd, I run the usual stty com2 115200 and
set tty com2, and then boot. The kernel messages show up fine, and
then the output just stops:

com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo
com2: console
[...]
root on rd0a swap on rd0b dump on rd0b
erase ^?, werase

Every time, it wedges up at this spot. The console still works for
kernel messages though, if I unplug the ipmi virtual cd, I see that:

erase ^?, werasesd2 detached
scsibus1 detached
umass0 detached

I tried installing it via the kvm head and configuring the serial
console for the installed OS, same problem, it spews all the kernel
messages and then:

com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo
com2: console
[...]
root on sd2a (b158108c15640a28.a) swap on sd2b dump on sd2b
Automatic boot i

Wedges up. Again, the kernel still works, if I plug in the virtual cd:

utomatic boot iumass0 at uhub2 port 1 configuration 1 interface 0 Ours
Technology product 0x rev 2.00/2.00 addr 4
umass0: using SCSI over Bulk-Only
scsibus3 at umass0: 2 targets, initiator 0

I see the messages. Interesting, the box ends up starting the network
and lets me log in via ssh, and when I do, there are *no* getty
processes at all. And if I try to reboot from the ssh login, ssh stops
responding, but it never reboots and I have to hard reset it.

It looks like as soon as userspace touches the console, it freezes up.
Nothing from userspace ever gets printed, and I can't type anything on
it. But kernel messages still show up.

The BIOS has the option to either enable or disable legacy redirection
after POST, I've tried it both ways with no difference. I can boot linux
and use the ipmi serial console just fine, seeing both the kernel output
and userspace output, getting a login prompt, and interacting with no
issues, so the underlying SOL is functional.

Any ideas what's going on or what I could try to fix or debug it?

Thanks much...

Re: IPMI SOL serial console wedges

[Paul B. Henson]

On Sun, Nov 24, 2013 at 09:54:41AM +0100, Sebastian Benoit wrote:

 in the bios, you can set the onboard serial ports irq to some higher value.
 that way, the ipmi console will become com0.
 
 (not tried on that board, only on newer supermicros that dont have the
 serial port on the outside anymore).

Hmm, no joy. I tried disabling both the physical serial ports, but the
SOL port stayed on the com2 ioport/irq. That completely broke the
bootloader, as it showed only com0, and when I told it to use it
presumably it tried the standard com0 ioport/irq. I tried setting the
bios to use the com1 ioport/irq for com0 and the com2 for com1 in the
hopes the SOL port would notice their was a conflict and use something
else, but nope, it still didn't work. I don't see anyway to set the SOL
port ioport/irq, I fear it's hardcoded.

It's weird that the kernel uses com2 as a console fine but then userland
borks it. The SOL port works fine as a console when I boot linux on the
box, so either there's an openbsd bug with it or linux must be
implementing some workaround for a problem.

Thanks anyway...


 Paul B. Henson(hen...@acm.org) on 2013.11.24 00:16:52 -0800:
  I've got a supermicro X9SCL-F board with ipmi support, and I'm trying to
  use it for the serial console. It shows up as a third com port. After
  booting the latest install cd, I run the usual stty com2 115200 and
  set tty com2, and then boot. The kernel messages show up fine, and
  then the output just stops:
  
  com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo
  com2: console
  [...]
  root on rd0a swap on rd0b dump on rd0b
  erase ^?, werase
  
  Every time, it wedges up at this spot. The console still works for
  kernel messages though, if I unplug the ipmi virtual cd, I see that:
  
  erase ^?, werasesd2 detached
  scsibus1 detached
  umass0 detached
  
  I tried installing it via the kvm head and configuring the serial
  console for the installed OS, same problem, it spews all the kernel
  messages and then:
  
  com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo
  com2: console
  [...]
  root on sd2a (b158108c15640a28.a) swap on sd2b dump on sd2b
  Automatic boot i
  
  Wedges up. Again, the kernel still works, if I plug in the virtual cd:
  
  utomatic boot iumass0 at uhub2 port 1 configuration 1 interface 0 Ours
  Technology product 0x rev 2.00/2.00 addr 4
  umass0: using SCSI over Bulk-Only
  scsibus3 at umass0: 2 targets, initiator 0
  
  I see the messages. Interesting, the box ends up starting the network
  and lets me log in via ssh, and when I do, there are *no* getty
  processes at all. And if I try to reboot from the ssh login, ssh stops
  responding, but it never reboots and I have to hard reset it.
  
  It looks like as soon as userspace touches the console, it freezes up.
  Nothing from userspace ever gets printed, and I can't type anything on
  it. But kernel messages still show up.
  
  The BIOS has the option to either enable or disable legacy redirection
  after POST, I've tried it both ways with no difference. I can boot linux
  and use the ipmi serial console just fine, seeing both the kernel output
  and userspace output, getting a login prompt, and interacting with no
  issues, so the underlying SOL is functional.
  
  Any ideas what's going on or what I could try to fix or debug it?
  
  Thanks much...
  
 
 -- 

Re: IPMI SOL serial console wedges

[Paul B. Henson]

On Sun, Nov 24, 2013 at 04:13:27PM -0500, Jiri B wrote:

 Supermicro IPMI is crap. Use normal serial console and add a power strip
 which you can manage via ethernet to poweroff/power cycle the server.

Well, I can't say it's the greatest implementation ever, but arguably it
doesn't seem much worse than on my Sun or IBM servers.

The exact same IPMI SOL works fine under linux and illumos, so it
doesn't necessarily seem an underlying fault in the implementation. I
guess I could try booting freebsd on it and see if that works.

If I used a regular serial console, I'd need a terminal server to access
it remotely, I'd rather just get this working and avoid the extra parts.

I see com2 was actually disabled by default until recently, were there
historically problems using com2 under openbsd? Does anybody have a
serial console working on a physical com2 port? Maybe it's not the SOL
at all but just being com2 that's causing the problem. Unfortunately
there doesn't seem to be any way to make the SOL port use a different
ioport/irq :(.

Re: IPMI SOL serial console wedges

[Paul B. Henson]

On 11/24/2013 2:00 PM, Theo de Raadt wrote:


Well, I can't say it's the greatest implementation ever, but arguably it
doesn't seem much worse than on my Sun or IBM servers.

[...]

You just cannot compare this to what Sun did, by (almost always) using
a seperate ethernet port. Probably still crap, but at least there was
some attempt to provide isolation.


Actually, my supermicro boxes have a separate dedicated IPMI ethernet 
port. Only some, not all, supermicro boards use that stupid bridged 
design that shares a single port for both the primary system ethernet 
and IPMI ethernet.



Is it still arguably fine?


What exactly are we arguing about again? My specific statement was not 
that IPMI was fine, nor even that it wasn't a piece of Swiss cheese; 
it was that the supermicro implementation is arguably not worse than 
other manufacturers. Of the articles referenced, the majority of the 
problems mentioned are in the base IPMI spec, while there were a couple 
of issues specific to supermicro, for the most part all vendors of IPMI 
hardware have the same underlying issues, and I'm sure other vendors 
have their own specific problems as well.


If you want to argue that overall IPMI is a sucky specification, and 
vendors in general have done a crap job of implementing it, you'll have 
to find somebody else to argue with, because I don't disagree with you. 
If you want to argue that supermicro in specific has done to a 
significant extent a crappier job, well, I think you need a bit more 
evidence.



The exact same IPMI SOL works fine under linux and illumos, so it
doesn't necessarily seem an underlying fault in the implementation. I
guess I could try booting freebsd on it and see if that works.


works fine


Yes, the IPMI SOL works fine. You can configure it to be a serial 
console, and the OS boots and functions correctly and successfully. 
Whether or not IPMI as a platform is a good idea or a secure 
implementation is really orthogonal to whether or not an OS can use a 
serial port.


Back on topic to my actual problem, it looks like the IPMI SOL com2 is 
actually using IRQ 10 rather than 5, which both linux and freebsd detect:


[2.324044] 00:0e: ttyS2 at I/O 0x3e8 (irq = 10) is a 16550A
uart2: 16550 or compatible port 0x3.8-0x3ef irq 10 on acpi0

I assume even if this were a physical serial port it would have the same 
problem, again nothing to do with IPMI. I'm going to try installing off 
the vga head again, and then tweaking the kernel to use irq 10 for com2 
rather than 5, and hopefully that will let me use the port as the serial 
console for the installed OS. If that works, presumably the installer 
could be fixed to work too, but that's probably more trouble than it's 
worth.


How come freebsd dynamically detects the correct irq, but openbsd has it 
hardcoded?



Similar issues exist on Dell BMC, documented elsewhere, you just need
to search.


Yep, IPMI kinda sucks. Not just on supermicro.


In fact it is unlikely that there are PC server-class
machines that have a safe primary ethernet port.  It is possible they
all have such problems built in (oops).


Are you talking about systems where the bmc shares a physical port with 
the OS? That's not what I have.



It is really amazing that so many people prefer to remain blissfully
unaware.


Of what? Potential issues with IPMI? No unawareness here… My boxes all 
have separate dedicated IPMI ports, all segregated onto a private 
network that sees no public traffic. Is it perfect? No. From a risk 
management/cost effectiveness assessment perspective for my specific 
deployment, is it better than not using it? Yes...

Re: IPMI SOL serial console wedges

[Paul B. Henson]

On Sun, Nov 24, 2013 at 12:16:52AM -0800, Paul B. Henson wrote:

 com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo
 com2: console
 [...]
 root on rd0a swap on rd0b dump on rd0b
 erase ^?, werase
 
 Every time, it wedges up at this spot. The console still works for
 kernel messages though, if I unplug the ipmi virtual cd, I see that:
[...] 
 It looks like as soon as userspace touches the console, it freezes up.

Well, it turns out the IPMI SOL com2 uses irq 10 for whatever reason
rather than irq 5. Both linux and freebsd successfully detect this:

[2.324044] 00:0e: ttyS2 at I/O 0x3e8 (irq = 10) is a 16550A
uart2: 16550 or compatible port 0x3.8-0x3ef irq 10 on acpi0

However, openbsd tries to use irq 5, which doesn't turn out so well. I'm
guessing the kernel doesn't use the irq, which is why those messages
still work, but userspace craps out when it tries to use the wrong irq.

I tweaked the kernel with config -e to change the irq for com2 to 10,
and it now works fine:

com2 at isa0 port 0x3e8/8 irq 10: ns16550a, 16 byte fifo
com2: console
[...]
root on sd2a (b158108c15640a28.a) swap on sd2b dump on sd2b
Automatic boot in progress: starting file system checks.
setting tty flags
pf enabled

I suppose the installer kernel could be fixed the same way, but at least
for this initial install it's not worth it, I'll just install with the
kvm head, fix the installed kernel, and then go serial from there.

Is there any particular reason openbsd can't dynamically detect the irq?

Re: IPMI SOL serial console wedges

[Paul B. Henson]

On Sun, Nov 24, 2013 at 04:10:23PM -0800, Paul B. Henson wrote:

 I suppose the installer kernel could be fixed the same way, but at least
 for this initial install it's not worth it, I'll just install with the
 kvm head, fix the installed kernel, and then go serial from there. 

Actually, it turned out to be pretty easy to fix the installer. The BIOS
redirection works for the initial bootloader, so you can use that to
switch the bootloader to use com2, then just boot '-c' to pop up the
inkernel editor to fix the irq, and then it all works from there...
Sweet..

Re: IPMI SOL serial console wedges

[Paul B. Henson]

On Mon, Nov 25, 2013 at 12:09:33PM +, Stuart Henderson wrote:

  How come freebsd dynamically detects the correct irq, but openbsd has it 
  hardcoded?
 
 linux and freebsd kernels use acpi to configure isa serial ports, openbsd
 uses static allocations.

Ah, ok; now that I know what's going on it's easy enough to fix, but it
was kinda confusing until I figured out the OS's that were working were
using a different interrupt.

 if you're wondering about the behaviour where you get the kernel
 messages, and it only stalls later, it just isn't looking at interrupts
 from the com port until that point.

Hmm, I saw kernel messages even after it stalled. All the initial kernel
messages were printed, then the first 16 chars from userspace, then
nothing else from userspace, but if I did something that resulted in kernel
messages (for example, plugging/unplugging the virtual usb cd) those
messages still came. 

 when enough chars have been sent to fill the buffer, we wait for an
 interrupt to say that it's ok to transmit again, but in your case,
 it's looking for the interrupt on the wrong pin so that interrupt
 it won't be seen.

Just userspace uses the interrupt? The kernel just pushes out characters
without waiting for a confirmation the fifo cleared?

Thanks much for the explanation...

Re: Intel 82574L vs 82579LM

[Paul B. Henson]

On Mon, Nov 25, 2013 at 04:30:36PM +0400, Alexander Pakhomov wrote:

 Both should not load CPU a lot. But that doesn't mean they wouldn't.
 Write here if notice intense interrupts CPU load. My OpenBSD 5.4 amd64
 laptop fail to handle 2 MB/s wifi due to some drivers issues (they
 load CPU up to 100% interrupts).  Additional info about interrupts
 load would be helpful.

I didn't find anything really conclusive. Evidentally the main
difference (http://supermicro.biz/support/faqs/faq.cfm?faq=11847) is
that the 82574L is a fully separate ethernet interface connected over
pcie, whereas the 82579LM is just a PHY that connects to the chipset
controller.

I found a couple anecdotal accounts that the 82579LM works better (such
as http://us.battle.net/d3/en/forum/topic/5151718781), but that's under
windows, and might just be driver issues.

I think I'm just going to set up the 82579LM for the high load initially
and see what happens; it will be easy enough to switch them around if I
want to try the other way later.

Re: Poor CARP Interface Performance with NAT

[Paul B. Henson]

On Tue, Jan 21, 2014 at 03:51:23PM -0800, Gabriel Kuri wrote:
 I am running obsd 5.4 as my NAT router. I decided to setup a second obsd
 box and run carp between the two for the external NATed interface (facing
 the ISP). After I setup everything and switched pf to NAT using the address
 on the carp interface, I'm seeing about 12Mbps - 13Mbps on the download, I
 have a 60Mbps pipe (down). When I switch pf back to NAT using the address
 on the physical interface, I get my full 60Mbps. Any ideas as to what I
 could be doing wrong that would limit performance through the carp
 interface to around 12Mbps - 13Mbps ?

You might want to try posting this to the pf mailing list:

http://www.benzedrine.cx/mailinglist.html

Maybe somebody there will have a suggestion?

L2TP VPN / pf

[Paul B. Henson]

I'm trying to get a L2TP VPN working using npppd; I think I'm most of the
way there but packets just aren't quite flowing. I'm not sure why, but I
think I might be missing something or misunderstanding something with pf.

I've got ipsec=YES and isakmpd_flags=-K in rc.conf.local,  and
/etc/ipsec.conf configured as:

-
ike passive esp transport \
proto udp from 96.251.22.154 to any port 1701 \
psk somesekretkeyhere
-

My /etc/npppd/npppd.conf is:

-
authentication LOCAL type local {
users-file /etc/npppd/npppd-users
}

tunnel L2TP_ipv4 protocol l2tp {
listen on 96.251.22.154
}

ipcp IPCP {
pool-address 10.128.120.2-10.128.120.254
dns-servers 10.128.0.4
}

interface pppx0 address 10.128.120.1 ipcp IPCP
bind tunnel from L2TP_ipv4 authenticated by LOCAL to pppx0
-

I currently have the following in pf.conf:

-
pass quick proto { esp, ah } from any to any
pass in quick on em1 proto udp from any to 96.251.22.154 port {500, 4500,
1701} keep state
set skip on enc0
set skip on pppx0
-

I'm pretty sure I have the ipsec/npppd pieces correct, as I am successfully
able to connect to the VPN:

-
2014-02-26 15:35:01:NOTICE: l2tpd ctrl=2 logtype=Started RecvSCCRQ
from=134.71.203.230:644
68/udp tunnel_id=2/6 protocol=1.0 winsize=4 hostname=Dogbert vendor=(no
vendorname) firm=0
000
2014-02-26 15:35:01:INFO: l2tpd ctrl=2 SendSCCRP
2014-02-26 15:35:01:INFO: l2tpd ctrl=2 RecvSCCN
2014-02-26 15:35:01:INFO: l2tpd ctrl=2 SendZLB
2014-02-26 15:35:01:INFO: l2tpd ctrl=2 call=21438 RecvICRQ session_id=3388
2014-02-26 15:35:01:INFO: l2tpd ctrl=2 call=21438 SendICRP session_id=21438
2014-02-26 15:35:01:INFO: l2tpd ctrl=2 call=21438 RecvICCN session_id=3388
calling_number=
 tx_conn_speed=100 framing=async
2014-02-26 15:35:01:NOTICE: l2tpd ctrl=2 call=21438 logtype=PPPBind ppp=1
2014-02-26 15:35:01:INFO: ppp id=1 layer=base logtype=Started
tunnel=L2TP_ipv4(134.71.203.
230:64468)
2014-02-26 15:35:01:INFO: l2tpd ctrl=2 call=21438 SendZLB
2014-02-26 15:35:04:INFO: ppp id=1 layer=lcp logtype=Opened mru=1360/1360
auth=MS-CHAP-V2 
magic=c638067d/52525adf
2014-02-26 15:35:04:INFO: ppp id=1 layer=chap proto=mschap_v2
logtype=Success username=he
nson realm=LOCAL
2014-02-26 15:35:04:INFO: ppp id=1 layer=ipcp IP Address peer=0.0.0.0
our=10.128.120.160.
2014-02-26 15:35:04:INFO: ppp id=1 layer=base unhandled protocol ipv6cp,
32855(8057)
2014-02-26 15:35:04:INFO: ppp id=1 layer=base unhandled protocol acsp,
3(8235)
2014-02-26 15:35:04:INFO: ppp id=1 layer=ccp CCP is stopped
2014-02-26 15:35:04:INFO: ppp id=1 layer=ipcp logtype=Opened
ip=10.128.120.160 assignType=
dynamic
2014-02-26 15:35:04:NOTICE: ppp id=1 layer=base logtype=TUNNELSTART
user=henson duration
=4sec layer2=L2TP_ipv4 layer2from=134.71.203.230:64468 auth=MS-CHAP-V2
ip=10.128.120.160 
iface=pppx0
2014-02-26 15:35:04:NOTICE: ppp id=1 layer=base Using pipex=yes
-

However, from the VPN client I cannot ping 10.128.120.1, the server
endpoint, and from the server I cannot ping 10.128.120.160, the client
endpoint. When I try to ping the client, I can see traffic on the ethernet
interface:

16:07:56.431711 esp bart.pbhware.com 
host-134-71-203-230.allocated.csupomona.edu spi 0x04e9efec seq 32 len 148
(DF)
16:07:57.441732 esp bart.pbhware.com 
host-134-71-203-230.allocated.csupomona.edu spi 0x04e9efec seq 33 len 148
(DF)
16:07:58.451762 esp bart.pbhware.com 
host-134-71-203-230.allocated.csupomona.edu spi 0x04e9efec seq 34 len 148
(DF)

And on the enc0 interface:

16:07:52.391543 (authentic,confidential): SPI 0x04e9efec:
bart.pbhware.com.l2tp  host-134-71-203-230.allocated.csupomona.edu.51118:
l2tp:[LS](8/3432)Ns=21,Nr=65535[hdlc|][|l2tp]
16:07:53.401575 (authentic,confidential): SPI 0x04e9efec:
bart.pbhware.com.l2tp  host-134-71-203-230.allocated.csupomona.edu.51118:
l2tp:[LS](8/3432)Ns=22,Nr=65535[hdlc|][|l2tp]

Conversely, when I try to ping the server from the client:

16:09:42.857872 esp host-134-71-203-230.allocated.csupomona.edu 
bart.pbhware.com spi 0x506039f8 seq 96 len 148
16:09:42.857875 esp host-134-71-203-230.allocated.csupomona.edu 
bart.pbhware.com spi 0x506039f8 seq 96 len 148
16:09:43.855422 esp host-134-71-203-230.allocated.csupomona.edu 
bart.pbhware.com spi 0x506039f8 seq 97 len 148

16:09:43.855454 (authentic,confidential): SPI 0x506039f8:
host-134-71-203-230.allocated.csupomona.edu.51118  bart.pbhware.com.l2tp:
l2tp:[L](1/12490)[hdlc|][|l2tp]
16:09:44.855469 (authentic,confidential): SPI 0x506039f8:
host-134-71-203-230.allocated.csupomona.edu.51118  bart.pbhware.com.l2tp:
l2tp:[L](1/12490)[hdlc|][|l2tp]
16:09:45.855498 (authentic,confidential): SPI 0x506039f8:
host-134-71-203-230.allocated.csupomona.edu.51118  bart.pbhware.com.l2tp:
l2tp:[L](1/12490)[hdlc|][|l2tp]

Ideally, I would like to completely disable pf at this point to confirm that
is the problem, but unfortunately it is a production router and I can't
really do that 8-/.

Am I missing something in either the ipsec, npppd, or pf 

Re: L2TP VPN / pf

[Paul B. Henson]

 From: YASUOKA Masahiko
 Sent: Wednesday, February 26, 2014 8:46 PM

 set skip on pppx0 needs to be improved because npppd may use pppx1,
 pppx2 ...

Once I've got things working, I'm probably going to want to have more
explicit rules rather than skipping; if I understand correctly I can just
use the generic interface group pppx rather than a specific instance so it
won't matter which one a given VPN connection is using.

   sysctl net.pipex.enable=1

Hmm, yeah, that... I had updated my /etc/sysctl.conf with that change, but
the system had not been rebooted since I did that; and it does appear I
forgot to run it by hand 8-/. I could have sworn I manually tweaked it when
I updated the config, but when I double checked, it was not set, so clearly
I did not sigh. After enabling that setting, I can ping the client from
the server and vice versa, yay! It looks like ospfd isn't propagating the
route for the VPN addresses, so I can't talk to anything past the router,
presumably I need to update that config next.

  For this rule pass quick proto { esp, ah } from any to any, does it
really
  need to be any to any with no interface defined?
 
 I think it is required only from/to the listening address of L2TP.

Cool. Now that it's somewhat working I will go back and clean that up.

 In L2TP/IPsec, transport mode IPsec is used instead of tunnel mode.
 This means enc(4) is not used.  And de-capsulated L2TP packets are
 received on the same interface which receives IPsec packet.

Hmm, that's not what I'm seeing. On the regular WAN interface (em1), when a
connection is established, I see some initial isakmp packets, and after
that, the only packets on that interface are the esp protocol:

12:23:29.614487 host-134-71-203-161.allocated.csupomona.edu.isakmp 
bart.pbhware.com.isak
mp: isakmp v1.0 exchange ID_PROT
cookie: 3d028e2c298bc357- msgid:  len: 500
12:23:29.614490 host-134-71-203-161.allocated.csupomona.edu.isakmp 
bart.pbhware.com.isak
mp: isakmp v1.0 exchange ID_PROT
cookie: 3d028e2c298bc357- msgid:  len: 500
[...]
mp: isakmp v1.0 exchange QUICK_MODE encrypted
cookie: 3d028e2c298bc357-d01b5baa5e938551 msgid: 080e9e87 len: 60
12:23:30.696974 host-134-71-203-161.allocated.csupomona.edu.isakmp 
bart.pbhware.com.isak
mp: isakmp v1.0 exchange QUICK_MODE encrypted
cookie: 3d028e2c298bc357-d01b5baa5e938551 msgid: 080e9e87 len: 60
12:23:30.697094 esp host-134-71-203-161.allocated.csupomona.edu 
bart.pbhware.com spi 0x9
81e7ad1 seq 1 len 116
12:23:30.697096 esp host-134-71-203-161.allocated.csupomona.edu 
bart.pbhware.com spi 0x9
81e7ad1 seq 1 len 116

And on the enc0 interface, I do see traffic, in particular what appears to
be the l2tp communication:

12:23:31.599555 (authentic,confidential): SPI 0x981e7ad1:
host-134-71-203-161.allocated.cs
upomona.edu.63807  bart.pbhware.com.l2tp: l2tp:[TLS](0/0)Ns=0,Nr=0
*MSGTYPE(SCCRQ) *PROTO
_VER(1.0) *FRAMING_CAP(AS) *HOST_NAME(Dogbert) *ASSND_TUN_ID(15)
*RECV_WIN_SIZE(4) [|l2tp]
12:23:31.599714 (authentic,confidential): SPI 0x0173ea42:
bart.pbhware.com.l2tp  host-134
-71-203-161.allocated.csupomona.edu.63807: l2tp:[TLS](15/0)Ns=0,Nr=1
*MSGTYPE(SCCRP) *PROT
O_VER(1.0) *FRAMING_CAP(S) *HOST_NAME(bart.pbhware.com) *ASSND_TUN_ID(1)
FIRM_VER(1282) |.
..[|l2tp]
12:23:31.622116 (authentic,confidential): SPI 0x981e7ad1:
host-134-71-203-161.allocated.cs
upomona.edu.63807  bart.pbhware.com.l2tp: l2tp:[TLS](1/0)Ns=1,Nr=1
*MSGTYPE(SCCCN) [|l2tp
]
12:23:31.622181 (authentic,confidential): SPI 0x0173ea42:
bart.pbhware.com.l2tp  host-134
-71-203-161.allocated.csupomona.edu.63807: l2tp:[TLS](15/0)Ns=1,Nr=2
ZLB[|l2tp]
12:23:31.624720 (authentic,confidential): SPI 0x981e7ad1:
host-134-71-203-161.allocated.cs
upomona.edu.63807  bart.pbhware.com.l2tp: l2tp:[TLS](1/0)Ns=2,Nr=1
*MSGTYPE(ICRQ) *ASSND_
SESS_ID(4051) *CALL_SER_NUM(1) [|l2tp]

For the purpose of writing pf rules, I'd like to understand exactly what
interfaces the packets are traversing. It looks like initially there are
isakmp packets from the remote client to the server on the external WAN
interface, followed by encapsulated esp packets. At that point, it looks
like the packets are flowing through the enc0 interface, from the remote
client to the l2tp server. Once that tunnel is established, it looks like
the tunneled packets flow through the pppx0 interface. So from a firewall
perspective, I think I just need a minimal set of rules on the WAN interface
and the enc0 interface to allow the l2tp tunnel to get established, and then
restrict the actual VPN traffic on the pppx interfaces.

Thanks much for helping point out my obvious mistake with pipex :). I wish I
would have thought to double check that yesterday when I was beating my head
against the wall trying to figure out why it wasn't working...

Re: L2TP VPN / pf

[Paul B. Henson]

 From: YASUOKA Masahiko
 Sent: Thursday, February 27, 2014 5:44 PM
  In L2TP/IPsec, transport mode IPsec is used instead of tunnel mode.
  This means enc(4) is not used.  And de-capsulated L2TP packets are
  received on the same interface which receives IPsec packet.
 
  Hmm, that's not what I'm seeing. On the regular WAN interface (em1),
  when a
  connection is established, I see some initial isakmp packets, and after
  that, the only packets on that interface are the esp protocol:
 
 You're right.  I was confused, sorry.

When I shut down the VPN client, it does look like there are some
non-encapsulated packets coming from the client to the l2tp port on the WAN
interface:

17:52:50.434908 esp bart.pbhware.com 
host-134-71-203-251.allocated.csupomona.edu spi 0x0
d875c54 seq 23 len 68
17:52:50.512023 host-134-71-203-251.allocated.csupomona.edu.isakmp 
bart.pbhware.com.isak
mp: isakmp v1.0 exchange INFO encrypted
cookie: 5eaec242bc98f28f-83a0e65155bcbcf8 msgid: c52320cb len: 76
17:52:50.512028 host-134-71-203-251.allocated.csupomona.edu.isakmp 
bart.pbhware.com.isakmp: isakmp v1.0 exchange INFO encrypted
cookie: 5eaec242bc98f28f-83a0e65155bcbcf8 msgid: c52320cb len: 76
17:52:50.512030 host-134-71-203-251.allocated.csupomona.edu.isakmp 
bart.pbhware.com.isak
mp: isakmp v1.0 exchange INFO encrypted
cookie: 5eaec242bc98f28f-83a0e65155bcbcf8 msgid: 64cf721d len: 92
17:52:50.512032 host-134-71-203-251.allocated.csupomona.edu.isakmp 
bart.pbhware.com.isakmp: isakmp v1.0 exchange INFO encrypted
cookie: 5eaec242bc98f28f-83a0e65155bcbcf8 msgid: 64cf721d len: 92
17:52:50.961996 host-134-71-203-251.allocated.csupomona.edu.62139 
bart.pbhware.com.l2tp:
 l2tp:[TLS](6/24262)Ns=4,Nr=2 *MSGTYPE(CDN) *ASSND_SESS_ID(4330)
*RESULT_CODE(768/0 ) [|l2tp]
17:52:50.962000 host-134-71-203-251.allocated.csupomona.edu.62139 
bart.pbhware.com.l2tp: l2tp:[TLS](6/24262)Ns=4,Nr=2 *MSGTYPE(CDN)
*ASSND_SESS_ID(4330) *RESULT_CODE(768/0 ) [|l2

They result in warnings from npppd though:

2014-02-27 17:52:50:INFO: l2tpd Received from=134.71.203.251:62139: bad
control message: tunnelId=6 is not found.  mestype=CDN
2014-02-27 17:52:52:INFO: l2tpd Received from=134.71.203.251:62139: bad
control message: tunnelId=6 is not found.  mestype=CDN

So they're clearly not important. The client I'm testing right now is an
iPhone, maybe it's a bug on that side.

 Not at all.  I'll fix npppd to warn in the log when the pipex sysctl
 is not set.

Yeah, that's a great idea. I was running in debug mode, and it logged:

2014-02-27 17:37:44:NOTICE: ppp id=0 layer=base Using pipex=yes

If there would have been a warning right next to it that pipex was disabled
that would've been quite helpful in pointing out the forgotten step.

npppd ipcp pool address configuration

[Paul B. Henson]

According to the npppd.conf man page:

 pool-address address-range | address-mask [for dynamic | static]
 Specify the IP address space that is pooled for this IPCP
 setting.  The address space can be specified by address-range
 (e.g. 192.168.0.2-192.168.0.254) or address-mask (e.g.
 192.168.0.0/24) .  dynamic means the address space is reserved
 for dynamic allocation; static means the address space is
 reserved for static allocation.  The default is dynamic.  This
 option can be used multiple times.

However, if I try to specify an address-mask:

ipcp IPCP {
pool-address 10.128.120.0/24
dns-servers 10.128.0.4
allow-user-selected-address no
}

It says there's a syntax error:

2014-02-28 11:48:24:NOTICE: Starting npppd pid=31351 version=5.0.0
2014-02-28 11:48:24:WARNING: pptpd GRE protocol not allowed
2014-02-28 11:48:24:CRIT: /etc/npppd/npppd.conf:12: syntax error
2014-02-28 11:48:24:CRIT: /etc/npppd/npppd.conf:17: ipcp IPCP is not found
2014-02-28 11:48:24:CRIT: /etc/npppd/npppd.conf:18: interface pppx0 is not found

I had originally specified an address range:

ipcp IPCP {
pool-address 10.128.120.2-10.128.120.254
dns-servers 10.128.0.4
allow-user-selected-address no
}

This works, but it's rather confusing in that it shows a whole bunch of tiny
allocations rather than a contiguous one:

2014-02-28 11:53:08:INFO: ipcp=IPCP pool 
dyn_pool=[10.128.120.2/31,10.128.120.4/30,10.128.120.8/29,10.128.120.16/28,10.128.120.32/27,10.128.120.64/26,10.128.120.128/26,10.128.120.192/27,10.128.120.224/28,10.128.120.240/29,10.128.120.248/30,10.128.120.252/31,10.128.120.254/32]
 
pool=[10.128.120.2/31,10.128.120.4/30,10.128.120.8/29,10.128.120.16/28,10.128.120.32/27,10.128.120.64/26,10.128.120.128/26,10.128.120.192/27,10.128.120.224/28,10.128.120.240/29,10.128.120.248/30,10.128.120.252/31,10.128.120.254/32]

I thought maybe if I used the address-mask rather than a range this would
be cleaner.

Is the man page incorrect or am I specifying the CIDR address wrong? Assuming
I want to allocate 10.128.120.1 as the local tunnel endpoint, and the rest of
that /24 as VPN addresses, what's the best way to configure it?

Thanks...

npppd l2tp-require-ipsec option

[Paul B. Henson]

After getting the basic functionality of an L2TP VPN working with npppd,
I tried turning on the l2tp-require-ipsec option, as that seemed
desirable; I don't really want an l2tp session set up that's not
encapsulated in ipsec.

However, with that option on, the attempted VPN connection doesn't seem
to get to npppd. After the ipsec negotiation, I see the l2tp packets
from the client on enc0:

12:20:38.080921 (authentic,confidential): SPI 0x18fc9556:
host-134-71-203-13.allocated.csupomona.edu.55757 
bart.pbhware.com.l2tp: l2tp:[TLS](0/0)Ns=0,Nr=0 *MSGTYPE(SCCRQ)
*PROTO_VER(1.0) *FRAMING_CAP(AS) *HOST_NAME(Dogbert) *ASSND_TUN_ID(36)
*RECV_WIN_SIZE(4) [|l2tp]
12:20:42.116036 (authentic,confidential): SPI 0x18fc9556:
host-134-71-203-13.allocated.csupomona.edu.55757 
bart.pbhware.com.l2tp: l2tp:[TLS](0/0)Ns=0,Nr=0 *MSGTYPE(SCCRQ)
*PROTO_VER(1.0) *FRAMING_CAP(AS) *HOST_NAME(Dogbert) *ASSND_TUN_ID(36)
*RECV_WIN_SIZE(4) [|l2tp]

But from npppd:

2014-02-28 12:20:26:INFO: l2tpd Listening 96.251.22.154:1701/udp (L2TP
LNS) [L2TP_ipv4]
nothing...

It doesn't log anything, it seems like it's just not even seeing the
connection attempt. If I disable l2tp-require-ipsec, it works fine
again.

Am I missing something or not understanding what this option is for?

Thanks...

ospfd and L2VPN routes

[Paul B. Henson]

I'm currently setting up an L2TP VPN with npppd. I've got the VPN piece
working, and can send packets between the client and the openbsd box
running the vpn. However, I'm currently using ospfd for routing between
the rest of the network and the openbsd box, and it doesn't seem to be
pushing routes for the IP addresses in use by the clients.

So, after a couple VPN clients connect, there are pppx interfaces:

pppx0: flags=8051UP,POINTOPOINT,RUNNING,MULTICAST mtu 1360
description: henson
priority: 0
groups: pppx
inet 10.128.120.1 -- 10.128.120.82 netmask 0x

pppx1: flags=8051UP,POINTOPOINT,RUNNING,MULTICAST mtu 1360
description: henson
priority: 0
groups: pppx
inet 10.128.120.1 -- 10.128.120.121 netmask 0x

and the local routing tables know how to get to them:

DestinationGatewayFlags   Refs  Use   Mtu  Prio Iface
10.128.120.82  10.128.120.1   UH 0   10 - 4 pppx0
10.128.120.121 10.128.120.1   UH 0   63 - 4 pppx1

ospfd seems to know *something* about the /24 I've allocated to the VPN:

flags: * = valid, O = OSPF, C = Connected, S = Static
Flags  Prio Destination  Nexthop  
*C4 10.128.110.0/24  link#7
  4 10.128.120.43/32 10.128.120.1
  4 10.128.120.45/32 10.128.120.1
  4 10.128.120.82/32 10.128.120.1
  4 10.128.120.107/3210.128.120.1
  4 10.128.120.121/3210.128.120.1
  4 10.128.120.160/3210.128.120.1
  4 10.128.120.163/3210.128.120.1
  4 10.128.120.165/3210.128.120.1
  4 10.128.120.208/3210.128.120.1
  4 10.128.120.212/3210.128.120.1
  4 10.128.120.214/3210.128.120.1
  4 10.128.120.219/3210.128.120.1
  4 10.128.120.223/3210.128.120.1
  4 10.128.120.233/3210.128.120.1
  4 10.128.120.246/3210.128.120.1
  4 10.128.120.248/3210.128.120.1
*O   32 10.128.130.0/24  10.128.0.14

But it doesn't have the active ones marked as valid, and it's not pushing
them, so there's no traffic flow between the vpn client and the network.

I currently have ospfd set to:

redistribute default
redistribute connected

While I am pushing a default route, I also have lower priority null routes
set on the other network equipment:

ip route 10.0.0.0 255.0.0.0 Null0 254
ip route 172.16.0.0 255.240.0.0 Null0 254
ip route 192.168.0.0 255.255.0.0 Null0 254

So they will blackhole any address space not valid on the network.

Am I missing some configuration that will make ospfd push out routes to
the client VPN addresses?

Thanks...

Re: npppd l2tp-require-ipsec option

[Paul B. Henson]

On Fri, Feb 28, 2014 at 01:54:13PM -0800, Jeff Goettsch wrote:
 That's a known bug:
 
 http://www.openbsd.org/cgi-bin/man.cgi?query=npppdapropos=0sektion=0manpath=OpenBSD+Currentarch=i386format=html#end

Ah, I see; I hadn't actually looked at the npppd man page, only the
npppd.conf man page. The BUGS section for npppd.conf doesn't list that,
although it mentions a different caveat.

Thanks...

Re: ospfd and L2VPN routes

[Paul B. Henson]

On Sat, Mar 01, 2014 at 11:23:01AM +0900, YASUOKA Masahiko wrote:

 I'm not sure whether it works.  Can you try it by static route?

A static route on the network on the other side of the openbsd box? I'm
sure that would work; when I try to ping a box out in the network from
the vpn client, I can see the outbound pings traversing the link from
the openbsd box to the router on the other side:

19:24:43.669307 10.128.120.163  10.128.130.1: icmp: echo request
19:24:44.646823 10.128.120.163  10.128.130.1: icmp: echo request
19:24:45.644309 10.128.120.163  10.128.130.1: icmp: echo request
19:24:46.666878 10.128.120.163  10.128.130.1: icmp: echo request

The return packets are getting dropped due to the lack of a route, so if
I had a static route on the other side it would work, but I'd rather not
use static routes, ideally I can make ospfd dynamically advertise routes
as necessary.

 Also, if there is a network behind the vpn, you can assign a static ip
 address and netmask instead of assigning /32 dynamic address.  See
 npppd-users(5) and use framed-ip-address and framed-ip-netmask.

I'd prefer not to assign a static IP per user, I like the concept of
just having a dynamic pool and users just get whatever address out of
it. I'm not sure how the netmask would work for a point-to-point link?
How could it be anything but a /32? Or would the netmask be for the route
on the other side? Right now it looks like the client is setting a
route to 10.0.0.0/8 across the tunnel, that should actually be
10.128.0.0/16, would setting the netmask in npppd-users fix that remote
route? Can I set the netmask but still let the client get a dynamic IP?

 npppd setup the routes for configured pool addresses to reserve them.
 I think this is the reason why ospfd seems to know something.

If I scrub everything clean, there are no routes to 10.128.120 in ospfd:

# ospfctl show fib | grep 120

Now, if I connect a client, route monitor shows:

RTM_IFANNOUNCE: iface arrival/departure: len 26, if# 18, name pppx0, what: 
arrival
got message of size 96 on Fri Feb 28 19:33:20 2014
RTM_NEWADDR: address being added to iface: len 96, metric 0, flags:
sockaddrs: NETMASK,IFP,IFA,BRD
 255.255.255.255 pppx0 10.128.120.1 10.128.120.230
got message of size 120 on Fri Feb 28 19:33:20 2014
RTM_ADD: Add Route: len 120, priority 4, table 0, pid: 0, seq 0, errno 0
flags:UP,HOST
use:0   mtu:0expire:0 
locks:  inits: 
sockaddrs: DST,GATEWAY
 10.128.120.230 10.128.120.1

and ospf lists it:

# ospfctl show fib | grep 120
  4 10.128.120.230/3210.128.120.1

 many /32 routes show something wrong.

Why? Isn't each instance of pppx for the VPN a /32 route to the remote IP?

It seems like everything is working as it should, other than ospf *not*
advertising the /32 route it finds out about? If ospfd pushed that route
out, the VPN client would work correctly.

Hmm, also, route monitor shows the route being removed when the client
disconnects:

RTM_DELETE: Delete Route: len 120, priority 0, table 0, pid: 0, seq 0, errno 0
flags:UP,HOST,DONE
use:0   mtu:0expire:0 
locks:  inits: 
sockaddrs: DST,GATEWAY
 10.128.120.230 10.128.120.1
got message of size 26 on Fri Feb 28 19:35:21 2014
RTM_IFANNOUNCE: iface arrival/departure: len 26, if# 18, name pppx0, what: 
departure

but ospfd doesn't lose it:

# ospfctl show fib | grep 120
  4 10.128.120.230/3210.128.120.1

until I explicitly reload the fib:

# ospfctl fib reload
reload request sent.
# ospfctl show fib | grep 120

That doesn't seem right either.

Thanks...

Re: npppd ipcp pool address configuration

[Paul B. Henson]

On Sat, Mar 01, 2014 at 12:56:16PM +0900, YASUOKA Masahiko wrote:
 Currently the parser needs to surrounding the address-mask with double
 quote like below:
 
   pool-address 10.128.120.0/24

Ah, yes; that's much better:

2014-03-01 15:59:13:INFO: ipcp=IPCP pool dyn_pool=[10.128.120.0/24]
pool=[10.128.120.0/24]

 And also we can use
 
   pool-address 10.128.120.0:255.255.255.0

Yes, that also results in a nice clean pool:

2014-03-01 16:00:43:INFO: ipcp=IPCP pool dyn_pool=[10.128.120.0/24]
pool=[10.128.120.0/24]

I think I'll stick with the quotes for now, I like CIDR notation :).

I had actually looked a bit at the parser before I posted, nothing
initially jumped out, but now that you point it out as the culprit I see
that the pool address is defined as a STRING, and based on:

#define allowed_in_string(x) \  
(isalnum(x) || (ispunct(x)  x != '('  x != ')'  \
x != '{'  x != '}'  x != ''  x != ''  \
x != '!'  x != '='  x != '/'  x != '#'  \
x != ','))

The literal / is not allowed in a string, whereas the literal : is.

From a quick look, I don't really see anything that would be broken by
allowing a / in a non-quoted string? Would just removing the explicit
restriction on the / in the above define fix it, or I am missing
something more subtle?

 As the default, npppd doesn't use the local tunnel endpoint address
 and broadcast addresses in class network (10.0.0.0 and 10.255.255.255)
 for the clients.  Do you worry about 10.128.120.0 or 10.128.120.255 in
 this case?

Technically, given they are all /32 point to point links, you could actually
make the local side .0 and use everything up to .255 (I think), but
I'm ok with losing two addresses to conform to usual /24 semantics and
less risk of confusion at some point. Setting the pppx IP to 10.128.120.1
and the ipcp pool to 10.128.120.0/24 will work perfectly for me, thanks
much for the help.

Re: ospfd and L2VPN routes

[Paul B. Henson]

On Sat, Mar 01, 2014 at 01:48:06PM +0900, YASUOKA Masahiko wrote:
  on the other side? Right now it looks like the client is setting a
  route to 10.0.0.0/8 across the tunnel, that should actually be
  10.128.0.0/16, would setting the netmask in npppd-users fix that remote
  route? Can I set the netmask but still let the client get a dynamic IP?
 
 My answer was wrong.  Assigning statically or netmask to the client is
 not related the ospf problem, I'm sorry.

No worries, I appreciate the help :). I tried setting the netmask in
npppd-users, that didn't change the /8 route the iPhone client set. From
a little investigation, it doesn't look like there's any way to set the
client netmask for the l2tp vpn route? The client just does whatever it
wants it seems, whether to just assume a class based route (/8 in the
case of my 10.128 address) or some seem to just assume a /24 8-/. You'd
think defining the client netmask would be part of the protocol, but
unless I'm missing something, I guess it's not.

 npppd set a /32 route for a VPN client and delete it when the link
 down.
 
  Isn't each instance of pppx for the VPN a /32 route to the remote
  IP?
 
 You had 16 /32 routes.  Don't you mean you had 32 VPN clients
 actually, right?

I only had one or two test clients connected at a time. But it looks
like ospfd picks up the route when a VPN client connects, but then
doesn't drop it when it disconnects, so the routes pile up.

After reloading the fib with no vpn clients, there are no /32 routes:

# ospfctl  fib reload   
reload request sent.

# ospfctl show fib | grep 120

I connect a client and a route shows up (but isn't advertised to the other
ospf connected routers):

# ospfctl show fib | grep 120
  4 10.128.120.109/3210.128.120.1

I disconnect the client, it's still there:

# ospfctl show fib | grep 120
  4 10.128.120.109/3210.128.120.1

I reconnect the client, it receives a different IP, and there are now
two routes:

# ospfctl show fib | grep 120
  4 10.128.120.109/3210.128.120.1
  4 10.128.120.155/3210.128.120.1

Disconnect, still two routes:

# ospfctl show fib | grep 120
  4 10.128.120.109/3210.128.120.1
  4 10.128.120.155/3210.128.120.1

Definitely something not quite right with ospfd and npppd l2tp vpns.

Thanks...

Re: ospfd and L2VPN routes

[Paul B. Henson]

On Sat, Mar 01, 2014 at 07:41:10PM +0900, YASUOKA Masahiko wrote:

 I could repeat the problem.  ospfd seems not to be able to use routes
 set by npppd.  The problem seems to be come from pppx(4)'s behavior of
 its link state.
 
 Using tun(4) instead of pppx(4) avoid the problem.

If I switch npppd to use tun, after startup (before any clients even
connect) ospfd shows three routes:

# ospfctl show fib | grep 120
*56 10.128.120.0/24  127.0.0.1
  4 10.128.120.1/32  10.128.120.1
*56 10.128.120.1/32  127.0.0.1

Kind of odd ones though, routing the whole /24 to localhost? After a
client connects, it adds another /32 to it:

# ospfctl show fib | grep 120
*56 10.128.120.0/24  127.0.0.1
  4 10.128.120.1/32  10.128.120.1
*56 10.128.120.1/32  127.0.0.1
 56 10.128.120.12/32 10.128.120.1

And after the client disconnects, it goes away:

# ospfctl show fib | grep 120
*56 10.128.120.0/24  127.0.0.1
  4 10.128.120.1/32  10.128.120.1
*56 10.128.120.1/32  127.0.0.1

However, it's still not advertised by ospfd to other routers, so the
client still can't communicate with the network beyond the openbsd
router terminating the vpn connection.

So while tun seems to avoid the pppx problem of dangling routes, it
still doesn't solve the issue of ospfd not advertising them sigh.

pppx is more efficient than tun, right? It keeps packets in the kernel
rather than bouncing them through userspace?

No ospfd experts that would like to chime in :)? The vpn /32 routes
don't seem to be marked as either valid or connected...

Thanks...

Re: ospfd and L2VPN routes

[Paul B. Henson]

 From: YASUOKA Masahiko
 Sent: Wednesday, March 05, 2014 1:48 AM

 framed-ip-netmask in npppd-user to set the netmask of the route to
 the PPP link.  But it is not to set the client netmask (on iPhone).
 
 AFAIK to set the client netmask, DHCP inform can be used.

Hmm, I thought the VPN client picked up its IP address from IPCP? How would
you get DHCP involved? Also, it's not really the netmask of the client's IP
address, but the mask for the route from the client to the VPN'd network
that needs to be tweaked.

Re: ospfd and L2VPN routes

[Paul B. Henson]

 From: YASUOKA Masahiko
 Sent: Wednesday, March 05, 2014 3:20 AM

   % ospfctl show fib | grep 128
   *56 10.128.120.0/24  127.0.0.1
   *56 10.128.120.213/3210.0.0.1

Interesting, not only does it show a /24 route, it looks like it has it
marked as valid. Is this with pppx or tun? IIRC, when I tested tun ospfd
found a /24 route, but still didn't propagate it.

 And do you know why the ospfd doesn't use the whole /24?

No, not a clue :(. I tried posting an inquiry to the tech list in the hope
someone familiar with the guts of osfpd might comment, but no responses so
far.

 Even if tun(4) is used, packets are processed in-kernel.

Ah, I had thought  pipex was just for pppx, but now I see in the man page it
says pipex is used with tun(4) and pppx. What's the difference between tun
and pppx in the context of npppd then? Is there any particular reason to use
one versus the other for an L2TP VPN?

Thanks much.

npppd can't open /dev/pppx1

[Paul B. Henson]

I set up an L2TP VPN with npppd recently using pppx, and other than some
routing issues with ospfd it works great. I'm trying to add a second VPN
connection, but that doesn't seem to work using pppx.

With this config:

interface pppx0 address 10.128.120.1 ipcp IPCP_admin
interface pppx1 address 10.128.120.129 ipcp IPCP

bind tunnel from L2TP_ipv4 authenticated by LOCAL_admin to pppx0
bind tunnel from L2TP_ipv4 authenticated by LOCAL to pppx1

npppd won't start:

# npppd -d
2014-03-19 14:08:27:NOTICE: Starting npppd pid=28792 version=5.0.0
2014-03-19 14:08:27:WARNING: pptpd GRE protocol not allowed
2014-03-19 14:08:27:NOTICE: Load configuration
from='/etc/npppd/npppd.conf' successfully.
2014-03-19 14:08:27:INFO: pppx0 Started pppx
2014-03-19 14:08:27:ERR: pppx1 open(/dev/pppx1) failed: No such file or 
directory

If I switch to tun instead of pppx:

interface tun0 address 10.128.120.1 ipcp IPCP_admin
interface tun1 address 10.128.120.129 ipcp IPCP
bind tunnel from L2TP_ipv4 authenticated by LOCAL_admin to tun0
bind tunnel from L2TP_ipv4 authenticated by LOCAL to tun1

it works fine:

# npppd -d
2014-03-19 14:14:28:NOTICE: Starting npppd pid=3355 version=5.0.0
2014-03-19 14:14:28:WARNING: pptpd GRE protocol not allowed
2014-03-19 14:14:28:NOTICE: Load configuration
from='/etc/npppd/npppd.conf' successfully.
2014-03-19 14:14:28:INFO: tun0 Started ip4addr=10.128.120.1
2014-03-19 14:14:28:INFO: tun1 Started ip4addr=10.128.120.129

Is there any way to make two VPN connections work with pppx, or are you
stuck with tun for that scenario?

Thanks...

Re: npppd can't open /dev/pppx1

[Paul B. Henson]

D'oh, I finally realized I needed to go to /dev and MAKEDEV pppx1 8-/.

Now it's working fine. I had thought pppx was one of those magic
clonable devices that you didn't need to explicitly create, I guess I
was mistaken. When I was testing the vpn, there were pppx1 and pppx2
interfaces that showed up in ifconfig for the clients, which I guess led
me to believe I didn't have to do anything special to use pppx1 in the
npppd config.

Thanks, and sorry for the noise.


On Wed, Mar 19, 2014 at 02:29:35PM -0700, Paul B. Henson wrote:
 I set up an L2TP VPN with npppd recently using pppx, and other than some
 routing issues with ospfd it works great. I'm trying to add a second VPN
 connection, but that doesn't seem to work using pppx.
 
 With this config:
 
 interface pppx0 address 10.128.120.1 ipcp IPCP_admin
 interface pppx1 address 10.128.120.129 ipcp IPCP
 
 bind tunnel from L2TP_ipv4 authenticated by LOCAL_admin to pppx0
 bind tunnel from L2TP_ipv4 authenticated by LOCAL to pppx1
 
 npppd won't start:
 
 # npppd -d
 2014-03-19 14:08:27:NOTICE: Starting npppd pid=28792 version=5.0.0
 2014-03-19 14:08:27:WARNING: pptpd GRE protocol not allowed
 2014-03-19 14:08:27:NOTICE: Load configuration
 from='/etc/npppd/npppd.conf' successfully.
 2014-03-19 14:08:27:INFO: pppx0 Started pppx
 2014-03-19 14:08:27:ERR: pppx1 open(/dev/pppx1) failed: No such file or 
 directory
 
 If I switch to tun instead of pppx:
 
 interface tun0 address 10.128.120.1 ipcp IPCP_admin
 interface tun1 address 10.128.120.129 ipcp IPCP
 bind tunnel from L2TP_ipv4 authenticated by LOCAL_admin to tun0
 bind tunnel from L2TP_ipv4 authenticated by LOCAL to tun1
 
 it works fine:
 
 # npppd -d
 2014-03-19 14:14:28:NOTICE: Starting npppd pid=3355 version=5.0.0
 2014-03-19 14:14:28:WARNING: pptpd GRE protocol not allowed
 2014-03-19 14:14:28:NOTICE: Load configuration
 from='/etc/npppd/npppd.conf' successfully.
 2014-03-19 14:14:28:INFO: tun0 Started ip4addr=10.128.120.1
 2014-03-19 14:14:28:INFO: tun1 Started ip4addr=10.128.120.129
 
 Is there any way to make two VPN connections work with pppx, or are you
 stuck with tun for that scenario?
 
 Thanks...

npppd with two pppx interfaces causes kernel panic

[Paul B. Henson]

After successfully setting up an L2TP VPN with npppd and pppx, I tried
to add a second VPN subnet with a different authentication base. I was
working remotely, and after starting npppd in debug mode:

bash-4.2# npppd -d
2014-03-19 14:41:50:NOTICE: Starting npppd pid=32407 version=5.0.0
2014-03-19 14:41:50:WARNING: pptpd GRE protocol not allowed
2014-03-19 14:41:51:NOTICE: Load configuration
from='/etc/npppd/npppd.conf' successfully.
2014-03-19 14:41:51:INFO: pppx0 Started pppx
2014-03-19 14:41:51:INFO: pppx1 Started pppx
2014-03-19 14:41:51:INFO: Listening /var/run/npppd_ctl (npppd_ctl)
2014-03-19 14:41:51:INFO: ipcp=IPCP_admin pool
dyn_pool=[10.128.120.0/25] pool=[10.128.120.0/25]
2014-03-19 14:41:51:INFO: ipcp=IPCP pool dyn_pool=[10.128.120.128/25]
pool=[10.128.120.128/25]
2014-03-19 14:41:51:INFO: Loading pool config successfully.

the box stopped responding :(. When I got on site, it was frozen and
nonresponsive. I rebooted, and on the way back up it panic'd when
starting npppd:

starting early daemons: syslogd pflogd named ntpd isakmpd npppd.
uvm_fault(0xfe812f620e00, 0x30, 0, 1) - e
fatal page fault in supervisor mode
trap type 6 code 0 rip 81385b40 cs 8 rflags 10257 cr2  30 cpl 0
rsp 8000221fdd38
panic: trap type 6, code=0, pc=81385b40
Starting stack trace...
panic() at panic+0xf5
trap() at trap+0x7f1
--- trap (number 6) ---
mtx_enter() at mtx_enter
VOP_KQFILTER() at VOP_KQFILTER+0x2b
kqueue_register() at kqueue_register+0x332
sys_kevent() at sys_kevent+0x115
syscall() at syscall+0x249
--- syscall (number 270) ---
end of kernel
end trace frame: 0x11be0a5e, count: 250
0x11be006eca6a:

It then said Syncing disks and sat there for 30 minutes, at which
point I gave up, booted in single user, and disabled npppd.
Unfortunately I don't have a serial console logger at the moment, so
while I assume it did the same panic when I was working remotely I don't
have logs for it. This is a 5.4 box with a generic kernel, other than
using config -e to enable ipmi and change the irq for com2.

Any thoughts on this? Here is the npppd config that causes it to blow
up:

authentication LOCAL_admin type local {
users-file /etc/npppd/npppd-users
username-suffix @admin
}
authentication LOCAL type local {
users-file /etc/npppd/npppd-users
}

tunnel L2TP_ipv4 protocol l2tp {
listen on 96.251.22.154
# l2tp-require-ipsec yes # buggy, doesn't work currently
}

ipcp IPCP_admin {
pool-address 10.128.120.0/25
dns-servers 10.128.0.4
allow-user-selected-address no
}
ipcp IPCP {
pool-address 10.128.120.128/25
dns-servers 10.128.0.4
allow-user-selected-address no
}

interface pppx0 address 10.128.120.1 ipcp IPCP_admin
interface pppx1 address 10.128.120.129 ipcp IPCP

bind tunnel from L2TP_ipv4 authenticated by LOCAL_admin to pppx0
bind tunnel from L2TP_ipv4 authenticated by LOCAL to pppx1

Re: npppd with two pppx interfaces causes kernel panic

[Paul B. Henson]

On Thu, Mar 20, 2014 at 10:22:51AM +0900, YASUOKA Masahiko wrote:

 pppx will be fixed.

Great :). This is a known bug then? Should I just keep an eye on the
changelog for mention of pppx changes to tell when it's safe to try
again?

 You can use tun(4) instead if you want to use multiple interfaces for
 that purpose.

Yes, I switched to tun for now pending the ability to have multiple pppx
interfaces defined. It was a rather big surprise for the box to
disappear on me while I was working with it, I don't have any out of
band access to it so it was offline until I got to it sigh.

Thanks...

Re: npppd with two pppx interfaces causes kernel panic

[Paul B. Henson]

 From: YASUOKA Masahiko
 Sent: Wednesday, March 19, 2014 9:44 PM

  Should I just keep an eye on the changelog for mention of pppx
  changes to tell when it's safe to try again?
 
 Sorry I cannot understand the point of this question.

Sorry to be confusing; I switched to tun because of this bug, but I would
prefer to use pppx once it has been fixed. The question was regarding how to
know when the bug has been fixed. As I am unaware of any publicly accessible
bug tracking system for openbsd, I was thinking of just watching the change
log (http://www.openbsd.org/plus.html) for any mention of pppx.

Thanks.

Re: npppd with two pppx interfaces causes kernel panic

[Paul B. Henson]

 From: Jonathan Gray
 Sent: Thursday, March 20, 2014 3:36 AM

 The following diff prevents the panic here:

Interesting, given the XXX, it seems somebody was already a little
suspicious of this section :).

From a cursory glance, it seems pppx_dev_lookup is supposed to return data
about a particular instance if it is in use, or NULL otherwise. So it seems
for some reason pppxclose is being called for device that isn't open? While
avoiding a panic is a plus :), it seems there is perhaps a logic error
somewhere else resulting in an attempt to close a device that isn't open?
 
 Index: if_pppx.c
 ==
 =
 RCS file: /cvs/src/sys/net/if_pppx.c,v
 retrieving revision 1.26
 diff -u -p -r1.26 if_pppx.c
 --- if_pppx.c 19 Oct 2013 14:46:30 -  1.26
 +++ if_pppx.c 20 Mar 2014 10:21:04 -
 @@ -590,7 +590,8 @@ pppxclose(dev_t dev, int flags, int mode
 
   rw_enter_write(pppx_devs_lk);
 
 - pxd = pppx_dev_lookup(dev);
 + if ((pxd = pppx_dev_lookup(dev)) == NULL)
 + return (ENXIO);
 
   /* XXX */
   while ((pxi = LIST_FIRST(pxd-pxd_pxis)))

skylake Xeon, C232 chipset, i210-AT ethernet

[Paul B. Henson]

I'm about to build a server with a supermicro X11SSL-F motherboard and a
Xeon E3-1240L v5 processor. The SATA ports should be AHCI compliant, and
it looks like the i210-AT ethernet is supported by the em driver, so I
think everything should work ok. But it's pretty new stuff, so I wanted
to check and see if anybody was aware of any problems or issues with
OpenBSD 5.8 and the latest Intel processors and chipsets before I pulled
the trigger on it.

Thanks much!

Re: Supermicro X11SSL-F freezes probing USB 3

[Paul B. Henson]

On Mon, Mar 28, 2016 at 03:06:39PM -0400, Sonic wrote:

> If I wait long enough the install will finally finish booting but the
> keyboard (no ps2 ports) doesn't work.

Could I trouble you to be more specific as to the duration of "long
enough" :)? I think my patience ran out after about 15-20 minutes. So it
eventually boots without disabling xhci, but the USB doesn't work in the
end anyway? I'm installing via an IPMI virtual serial port so the lack
of keyboard isn't really an issue for me, I can live without USB but as
the box won't be going live for a few weeks I thought I'd see if any
devs wanted me to try anything on it before I just moved forward without
USB support. I've got -current set up to ready to patch and compile to
test stuff on it if I can. It would be nice to get it working for
situations like yours where it's needed.

I booted a FreeBSD 10.2 livecd on it, and that initialized the xhci
chipset fine and usb devices seem to work ok. I tried to compare the
drivers, they share a bit in common but they're also quite different and
it doesn't help that I'm not really a low level driver guy 8-/. I'm sure
the new Skylake stuff just needs some minor tweak to make it happy.

Thanks...

Re: Supermicro X11SSL-F freezes probing USB 3

[Paul B. Henson]

On Tue, Mar 29, 2016 at 10:46:15PM -0400, Sonic wrote:

> The IPMI is part of Dell's iDRAC stuff and the only thing I've found
[...]
> may be the iDRAC license level as well, anything above the "basic"
> level, providing a limited feature set, requires purchasing a license

Eeew. We've got some HP gear that requires an extra cost license to make
the remote kvm gui head work past the bootloader which is ridiculous
(but technically, I don't think remote kvm is part of the base IPMI
standard), but the IPMI SOL serial port??? That's just crazy. I've never
used Dell and never will for servers; desktops/notebooks, sure, but
servers? Nah. Sun gear was pretty good until Oracle killed them off, we
used IBM for a while until they sold it off to Lenovo and policy
wouldn't let us buy from a non-US company (like the gear itself doesn't
come from China anyway). Right now we're using HP at my dayjob and it's
working out ok. I pretty much use supermicro for personal gear and
sidejobs, it's generally good stuff. At least my IPMI SOL port works :).

Good luck :).

Re: Supermicro X11SSL-F freezes probing USB 3

[Paul B. Henson]

On Wed, Mar 30, 2016 at 03:34:25PM -0400, Sonic wrote:

> Ahha! Who would have thought... com0 was the ticket. Thanks much!

Sweet, glad to hear you got it working. Usually the IPMI SOL comes after
the physical serial ports, I've never seen it be the first one. But hey,
it's Dell :).

Maybe now that 5.9 is out (a month early, nice, just in time for my new
box) one of the devs will have time to take a look at the skylake
usb 3 issues.

Supermicro X11SSL-F freezes probing USB 3

[Paul B. Henson]

I just put together a new server with a Supermicro X11SSL-F motherboard
and a Xeon E3-1240L v5 processor, and was trying to install openbsd 5.8
on it. The install cd freezes while booting after it probes the USB 3
devices:

>>> xhci probe won
xhci0 at pci0 dev 20 function 0 "Intel 100 Series xHCI" rev 0x31: msi
>>> probing for usb*
>>> usb probe returned 1
>>> usb probe won
usb0 at xhci0: USB revision 3.0
>>> probing for uhub*
>>> uhub probe returned 10
>>> uhub probe won
uhub0 at usb0 "Intel xHCI root hub" rev 3.00/1.00 addr 1
[system freezes here]


I also tried the latest snapshot install cd, same problem. If I disable
xhci, the installer boots successfully, although I haven't actually
tried installing yet. I don't really need usb on this box, so I'm not
really concerned if it's not going to work. It's not going into
production for a few weeks though, so if anybody's interested in looking
at why it's broken I could provide further details or test possible
fixes. Here's a dmesg of the snapshot install kernel booted without
xhci:

OpenBSD 5.9-current (RAMDISK_CD) #1737: Sun Mar  6 19:18:13 MST 2016
   
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/RAMDISK_CD
   
real mem = 16976416768 (16189MB)
   
avail mem = 16460062720 (15697MB)   
   
User Kernel Config  
   
UKC> do\^H \^Hiso\^H \^Hable xhci   
   
 98 xhci* disabled  
   
 UKC> quit  

 Continuing...  

 mainbus0 at root   

 bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x7fb95000 (59 entries)   

 bios0: vendor American Megatrends Inc. version "1.0b" date
 12/29/2015  
 bios0: Supermicro Super Server 

 acpi0 at bios0: rev 2  

 acpi0: tables DSDT FACP APIC FPDT FIDT SPMI MCFG HPET SSDT LPIT
 SSDT SSDT SSDT DBGP DBG2 SSDT SSDT UEFI SSDT DMAR EINJ ERST BERT
 HEST 
 acpimadt0 at acpi0 addr 0xfee0: PC-AT compat   

 cpu0 at mainbus0: apid 0 (boot processor)  

 cpu0: Intel(R) Xeon(R) CPU E3-1240L v5 @ 2.10GHz, 2100.73 MHz  

 cpu0:
 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT

 cpu0: 256KB 64b/line 8-way L2 cache

 cpu0: apic clock running at 24MHz  

 cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1, IBE   

 cpu at mainbus0: not configured

 cpu at mainbus0: not configured

 cpu at mainbus0: not configured

 cpu at mainbus0: not configured

 cpu at mainbus0: not configured

 cpu at mainbus0: not configured

 cpu at mainbus0: not configured

 ioapic0 at mainbus0: apid 2 pa 0xfec0, version 20, 24 pins 

 acpiprt0 at acpi0: bus 0 (PCI0)

 acpiprt1 at acpi0: bus 1 (PEG0)

 acpiprt2 at acpi0: bus -1 (PEG1)   

 acpiprt3 at acpi0: bus -1 (PEG2)   

 acpiprt4 at acpi0: bus 2 (RP09)

 acpiprt5 at acpi0: bus 3 (RP10)  

Re: Supermicro X11SSL-F freezes probing USB 3

[Paul B. Henson]

On Tue, Mar 29, 2016 at 04:55:05PM -0400, Sonic wrote:

> Unfortunately that option isn't available for me. The IPMI SOL on this
> Dell stops forwarding the console once the system boots.

The usb keyboard should still work when the bootloader is running,
that's being handled by the BIOS. You just need to determine what
port/baud rate the IPMI serial port is on your system, and when the
bootloader shows up, just type for example:

stty com1 115200
set tty com1

That will switch the bootloader to directly use the serial port, and
when you boot the OS, it will do so as well. After these two commands,
just 'boot -c' as usual to disable xhci, and continue on the serial
port. Real servers don't need heads or keyboards ;).

At least on my supermicro box, the default bios setting also allows me
to type that on the IPMI serial console as well, in addition to the boot
up messages it also forwards the bootloader by default, it doesn't stop
forwarding until the OS itself loads.

If you install over the serial console, the OS will by default be
installed to use it too. So maybe you don't need that keyboard after all
:).

no SDRs IPMI disabled?

[Paul B. Henson]

I just installed 5.9 on a Supermicro X11SSL-F board, and tried to enable
the ipmi driver. During boot, it shows:

ipmi0 at mainbus0: version 2.0 interface KCS iobase 0xca2/2 spacing 1
iic0: skipping sensors to avoid ipmi0 interactions
ipmi0: get header fails
ipmi0: no SDRs IPMI disabled
ipmi at mainbus0 not configured

Any suggestions on how to make this work? The full dmesg is:


OpenBSD 5.9 (GENERIC.MP) #1888: Fri Feb 26 01:20:19 MST 2016
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 16976416768 (16189MB)
avail mem = 16457711616 (15695MB)
User Kernel Config
UKC> enable ipmi
401 ipmi0 enabled
UKC> quit
Continuing...
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x7fb95000 (59 entries)
bios0: vendor American Megatrends Inc. version "1.0b" date 12/29/2015
bios0: Supermicro Super Server
acpi0 at bios0: rev 2
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP APIC FPDT FIDT SPMI MCFG HPET SSDT LPIT SSDT SSDT SSDT 
DBGP DBG2 SSDT SSDT UEFI SSDT DMAR EINJ ERST BERT HEST
acpi0: wakeup devices PEGP(S4) PEG0(S4) PEGP(S4) PEG1(S4) PEGP(S4) PEG2(S4) 
PXSX(S4) RP09(S4) PXSX(S4) RP10(S4) PXSX(S4) RP11(S4) PXSX(S4) RP12(S4) 
PXSX(S4) RP13(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E3-1240L v5 @ 2.10GHz, 2100.85 MHz
cpu0: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 24MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU E3-1240L v5 @ 2.10GHz, 2100.00 MHz
cpu1: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Xeon(R) CPU E3-1240L v5 @ 2.10GHz, 2100.00 MHz
cpu2: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Xeon(R) CPU E3-1240L v5 @ 2.10GHz, 2100.00 MHz
cpu3: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 1 (application processor)
cpu4: Intel(R) Xeon(R) CPU E3-1240L v5 @ 2.10GHz, 2100.00 MHz
cpu4: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT
cpu4: 256KB 64b/line 8-way L2 cache
cpu4: smt 1, core 0, package 0
cpu5 at mainbus0: apid 3 (application processor)
cpu5: Intel(R) Xeon(R) CPU E3-1240L v5 @ 2.10GHz, 2100.00 MHz
cpu5: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT
cpu5: 

Re: Supermicro X11SSL-F freezes probing USB 3

[Paul B. Henson]

On Tue, Mar 29, 2016 at 07:06:41PM -0400, Sonic wrote:
> On Tue, Mar 29, 2016 at 6:15 PM, Paul B. Henson <hen...@acm.org> wrote:
> > stty com1 115200
> > set tty com1
> 
> Yes, tried that with no luck, SOL still stops forwarding. The box does

Hmm, that sounds broken. Are you sure you've got the right serial port
and baud rate? Once you switch the boot loader to serial, it's no longer
a matter of "forwarding", it's direct serial access as far as the
bootloader/OS is concerned. The BIOS forwarding piece is out of the
picture. Unless your IPMI serial port implementation is broken you've
probably got the wrong settings. Double check in the BIOS what the IPMI
serial port settings are (which port, speed, etc) and make sure they
match what you tell the bootloader.

> rely on ssh for everything. But there's always some possible problem
> that it would be nice to be able to plug in a keyboard and monitor to
> work with.

Can't you use the IPMI virtual head? I can't remember the last time I
used a physical anything with a server. Rack and forget... Unless a
power supply blows.

Re: what all touches the carp demote counter?

[Paul B. Henson]

On Mon, Oct 10, 2016 at 09:43:56PM -0300, R0me0 *** wrote:

> Did you adjust advskew value on the machine you want to be Backup ?

Yes, the backup has an advskew of 5 and the primary an advskew of 1. As
I mentioned, when I first configured the interfaces by hand the two
systems properly negotiated master/backup roles, it was only after I
rebooted the one that was supposed to be primary on this interface that
it came up as backup, and I traced it to the fact the the carp demote value
was set to 2. When I manually changed the carp demote value to 0, the
system once again pre-empted the master role on the interface.

I'm just not sure what is twiddling with the carp demotion value. Unless
ospdf does it by default? The man page for the config file reads like it
would only do it if you explicitly include the demote keyword in the
area or interface section.

Thanks for the suggestion though.

Re: what all touches the carp demote counter?

[Paul B. Henson]

Arg, I'm still having issues with the carp demote counter. I disabled
ospfd for now, but something is still changing it. After a reboot
without ospfd, the counter is changing between 0 and 1:

bash-4.3# ifconfig -g carp
carp: carp demote count 1

bash-4.3# ifconfig -g carp
carp: carp demote count 0

bash-4.3# ifconfig -g carp
carp: carp demote count 1

bash-4.3# ifconfig -g carp
carp: carp demote count 0

And the carp interface is flapping:

Oct 14 13:17:17 lisa /bsd: carp0: state transition: BACKUP -> MASTER
Oct 14 13:17:23 lisa /bsd: carp0: state transition: MASTER -> BACKUP
Oct 14 13:17:43 lisa /bsd: carp0: state transition: BACKUP -> MASTER
Oct 14 13:17:49 lisa /bsd: carp0: state transition: MASTER -> BACKUP
Oct 14 13:18:08 lisa /bsd: carp0: state transition: BACKUP -> MASTER

There's not too much running; smtpd, sshd, npppd, dhcpd. Any suggestions
as to what might be screwing with the carp demote value?

Thanks...


root 1  0.0  0.0   440   520 ??  Is 1:14PM0:01.01 /sbin/init
root 21696  0.0  0.0  1044  1296 ??  Isp1:14PM0:00.00 syslogd: 
[priv] (syslogd)
_syslogd 22103  0.0  0.0  1044  1388 ??  Sp 1:14PM0:00.07 
/usr/sbin/syslogd
_pflogd   5335  0.0  0.0   684   400 ??  Sp 1:14PM0:00.02 pflogd: 
[running] -s 160 -i pfl
root 27252  0.0  0.0   620   600 ??  Is 1:14PM0:00.00 pflogd: 
[priv] (pflogd)
_ntp 16170  0.0  0.0   636  1472 ??  Isp1:14PM0:00.02 ntpd: dns 
engine (ntpd)
_ntp 15754  0.0  0.0   688  1540 ??  S I'm setting up a second router that's going to sit next to an existing
> one and become a redundant failover system. The current one is in
> production, and I've been converting some of the existing LAN subnets on it
> to use carp interfaces and making them primary and the new box
> secondary. I also set up a carp interface on the WAN side and made the
> new box primary for testing as that didn't exist before. That all
> worked fine when I set it up by hand, but when I rebooted the new box,
> the old box stayed primary for everything including the WAN interface,
> which I tracked down to the carp demote counter, which ended up at 2 on
> the new box after the reboot:
> 
> bash-4.3# ifconfig -g carp
> carp: carp demote count 2
> 
> After I manually decreased the demote counter by 2 back to 0 the WAN
> interface master switched back to the new box.
> 
> I'm not sure what's doing that at boot? I am running ospfd on the box,
> but I don't have any demote statements in my configuration. I'm also
> running npppd, but I don't see anything about that and carp demotion.
> What else might be setting carp demotion values?
> 
> Thanks...

Re: what all touches the carp demote counter?

[Paul B. Henson]

On Fri, Oct 14, 2016 at 01:27:42PM -0700, Paul B. Henson wrote:
> Arg, I'm still having issues with the carp demote counter. I disabled
> ospfd for now, but something is still changing it. After a reboot
> without ospfd, the counter is changing between 0 and 1:

Ah, I tracked it down. I had configured another carp interface on the
new system which didn't yet have a corresponding interface on the old
system. I have the carp interfaces configured with explicit peer
addresses rather than using multicast, and evidentally the inability to
send a packet to the peer was causing the other carp interface to
twiddle the global carp demote counter, which popped up once I cranked
up the carp log level:

Oct 14 15:21:48 lisa /bsd: carp: carp1 demoted group carp by -1 to 2 (< 
snderrors)
Oct 14 15:21:52 lisa /bsd: carp1: ip_output failed: 64
Oct 14 15:21:54 lisa /bsd: carp: carp1 demoted group carp by 1 to 3 (> 
snderrors)
Oct 14 15:21:55 lisa /bsd: carp1: ip_output failed: 64
Oct 14 15:22:14 lisa /bsd: carp: carp1 demoted group carp by -1 to 2 (< 
snderrors)
Oct 14 15:22:18 lisa /bsd: carp1: ip_output failed: 64
Oct 14 15:22:20 lisa /bsd: carp: carp1 demoted group carp by 1 to 3 (> 
snderrors)

It doesn't do this if I remove the carppeer and use the default multicast;
that's an unexpected side effect of configuring a carppeer that might be
worth documenting. A down carppeer on one interface can impact the
functionality of all carp interfaces on the system.

Re: what all touches the carp demote counter?

[Paul B. Henson]

On Tue, Oct 11, 2016 at 08:44:05AM +0200, mxb wrote:

> Master-Backup setup with pfsync in place, means that you synchronize
> states between boxes.  Then Master is rebooted, it becomes out-of-sync
> then it comes to states.  So until it is in sync with Backup (which
> became Master after reboot), it will not become Master.
> 
> This process is auto. Just need to wait.

I haven't set up pfsync yet, I need to upgrade the old box first. Right
now I'm just working with carp. Does pfsync fiddle with the carp
demotion value even if it's not configured?

Thanks...

what all touches the carp demote counter?

[Paul B. Henson]

I'm setting up a second router that's going to sit next to an existing
one and become a redundant failover system. The current one is in
production, and I've been converting some of the existing LAN subnets on it
to use carp interfaces and making them primary and the new box
secondary. I also set up a carp interface on the WAN side and made the
new box primary for testing as that didn't exist before. That all
worked fine when I set it up by hand, but when I rebooted the new box,
the old box stayed primary for everything including the WAN interface,
which I tracked down to the carp demote counter, which ended up at 2 on
the new box after the reboot:

bash-4.3# ifconfig -g carp
carp: carp demote count 2

After I manually decreased the demote counter by 2 back to 0 the WAN
interface master switched back to the new box.

I'm not sure what's doing that at boot? I am running ospfd on the box,
but I don't have any demote statements in my configuration. I'm also
running npppd, but I don't see anything about that and carp demotion.
What else might be setting carp demotion values?

Thanks...

Re: what all touches the carp demote counter?

[Paul B. Henson]

On Wed, Oct 12, 2016 at 08:37:59AM +0200, mxb wrote:

> But as R0me0 stated, you should probably re-check your configuration.

The configuration checked out. I rebooted a few more times, and I
couldn't reproduce the problem. I still have no idea why the carp
demotion counter was set to 2 the first time I rebooted. It doesn't seem
to be doing it anymore though. Thanks for all the suggestions though, it
helped to verify everything was set up right.

WARNING: symbol(icudt58_dat) size mismatch, relink your program

[Paul B. Henson]

I'm trying to compile openldap from ports under 6.1, and running it
fails with the error:

slapd:/usr/local/lib/libicuuc.so.12.0: /usr/local/lib/libicudata.so.12.0
: WARNING: symbol(icudt58_dat) size mismatch, relink your program

I see there was some dicussion of this back around April, but no
resolution, and I didn't see anything since then. Evidentally it impacts
anything that uses textproc/icu from what I could tell. I poked around
with it a bit but nothing jumped out as to why it's doing this. The
symbol seems to be defined in libicudata.so and accessed by libicuuc.so.
The actual object file in the distibution that contains it is
dynamically generated. I have the exact same version running ok on a
linux box so it doesn't seem to be an issue with the code itself.

Has anyone figured out what's going on with this code under openbsd
that's causing it to fail like this?

Thanks...

Re: WARNING: symbol(icudt58_dat) size mismatch, relink your program

[Paul B. Henson]

On Wed, Aug 02, 2017 at 05:37:40PM -0700, Paul B. Henson wrote:
> I'm trying to compile openldap from ports under 6.1, and running it
> fails with the error:
> 
> slapd:/usr/local/lib/libicuuc.so.12.0: /usr/local/lib/libicudata.so.12.0
> : WARNING: symbol(icudt58_dat) size mismatch, relink your program

I ended up checking out the 6.0 version of textproc/icu (57.1) into my 6.1
ports tree and compiling that, which seems to work fine. There must just
be some weird issue with 58.2 under OpenBSD.

Re: WARNING: symbol(icudt58_dat) size mismatch, relink your program

[Paul B. Henson]

On Thu, Aug 03, 2017 at 05:33:15PM -0400, Predrag Punosevac wrote:

> It is well known issue.
> 
> https://marc.info/?l=openbsd-misc=149271724912565=2
> 
> It seems to be benign at least for my use case.

Yah, I saw that discussion from back in April, but then it just stopped
with no resolution. I'm not sure what your use case is, but as far as I
can tell, it's preventing programs linked against libicuuc.so from
running? So not too benign for me 8-/. But fortunately downgrading to
the 6.0 version of the port seems to have worked around the issue.

Thanks...

Re: openldap port mdb support

[Paul B. Henson]

On Mon, Jul 10, 2017 at 07:34:11AM +, Stuart Henderson wrote:

> Feel free to try it, I believe the required patch to force MDB_WRITEMAP
> is still in there..but I don't think there were any major changes upstream
> since the last attempt so I wouldn't hold out too much hope for it working
> straight off.

Hmm, as you said, trying to use mdb resulted in crashes. My initial debugging
led to the cause of this as a NULL mdb environment, and ironically the
root cause of that turned out to be the OpenBSD specific MDB_WRITEMAP
patch 8-/.

if ( !(flags & MDB_WRITEMAP) ) {
Debug( LDAP_DEBUG_ANY,
LDAP_XSTRING(mdb_db_open) ": database \"%s\" does not 
have writemap. "
"This is required on systems without unified buffer 
cache.\n",
be->be_suffix[0].bv_val, rc, 0 );
goto fail;
}

There are two problems with it; first, it accesses the local flags variable
before it is initialized to mdb->mi_dbenv_flags shortly thereafter, so the
value checked is random and the if block nondeterministically triggers, and
second, it doesn't assign a failure value to rc before it jumps to fail, so
the function returns successfully but with a closed be, and the code keeps
going but later segfaults because of the NULL mdb environment.

I updated the patch and moved the check to be after the flags initialization:

flags = mdb->mi_dbenv_flags;

and added an assignment to rc on failure:

rc = MDB_INCOMPATIBLE;

I then tweaked the mdb test suite to always enable MDB_WRITEMAP, and so far
it's been running for 20 minutes with no errors, crashes, or failures.

Right now it's compiled "-O0 -ggdb", if everything keeps looking good, I'll
recompile it normally and do more testing.

Re: OpenBSDI 6.1 some Warnings when using OpenLDAP Tools

[Paul B. Henson]

On Wed, Aug 09, 2017 at 09:06:19AM +0200, Markus Rosjat wrote:

> this is more an info then a problem though since it seems to work.
> When I use the slap tool like slapcat I get a size mismatch warning like 
> this

Heh, we were just talking about that:

https://marc.info/?l=openbsd-misc=150199443929908=2

openldap port mdb support

[Paul B. Henson]

mdb has been disabled in the openldap port since it looks like
2015/02/16, I was wondering if anyone has tried it since then to see if
maybe the issues with it have been resolved? The other backends are
deprecated upstream, it would be nice to get mdb working under openbsd.

I'm going to try enabling it and running through the tests and see how
things turn out but I was just curious if anyone else had worked with it
in the past couple of years.

Thanks...

Re: ipmi driver broken

[Paul B. Henson]

On Wed, Jun 28, 2017 at 06:31:34PM -0400, Predrag Punosevac wrote:

> My understanding is that ipmi driver used by ipmitool is disabled
> intensionally due to the security problems. IPMI pose a grave security
> risk.

IPMI on the SP is available whether or not the openbsd driver is enabled
or in use; my understanding as to why it's disabled by default it that
it's not necessarily considered stable. I've never had an issue with it,
at least not for the limited use I make of it.

> As you probably know OpenBSD comes with its own sensoring
> framework. You probably want to check out

Yes; I actually want the ipmi driver loaded so it can supply data to
said framework:

hw.sensors.ipmi0.temp0=34.00 degC (System Temp), OK
hw.sensors.ipmi0.temp1=40.00 degC (Peripheral Temp), OK
hw.sensors.ipmi0.fan0=4875 RPM (FAN 1), OK
hw.sensors.ipmi0.fan1=3000 RPM (FAN 2), OK
hw.sensors.ipmi0.fan2=3150 RPM (FAN 3), OK
hw.sensors.ipmi0.fan3=5100 RPM (FAN 4), OK
hw.sensors.ipmi0.fan4=3300 RPM (FAN A), OK
hw.sensors.ipmi0.volt0=0.71 VDC (Vcore), OK
hw.sensors.ipmi0.volt1=3.23 VDC (3.3VCC), OK
hw.sensors.ipmi0.volt2=12.14 VDC (12V), OK
hw.sensors.ipmi0.volt3=1.53 VDC (VDIMM), OK
hw.sensors.ipmi0.volt4=4.99 VDC (5VCC), OK
hw.sensors.ipmi0.volt5=-12.49 VDC (-12V), OK
hw.sensors.ipmi0.volt6=3.17 VDC (VBAT), OK
hw.sensors.ipmi0.volt7=3.36 VDC (VSB), OK
hw.sensors.ipmi0.volt8=3.23 VDC (AVCC), OK
hw.sensors.ipmi0.indicator0=Off (Chassis Intru), OK

There's more sensor data available via the IPMI interface than the
kernel supplies without it. It's also useful to be able to view the SEL
without having to loop over the network to the SP management IP. On my
linux boxes I also use the ipmi hardware watchdog, but last time I tried
that on openbsd it just kept rebooting continuously 8-/. Guess that's
one of the parts that's not stable :), but I can't remember the last
time one of my openbsd boxes wedged up anyway.

Anyway, thanks for the thoughts; but I do still want a working ipmi :).
No biggie to add one line and recompile the kernel, but it would be nice
to get fixed. It's still disabled by default out of the box, you have to
explicitly reconfigure your kernel to enable it.

Re: ipmi driver broken

[Paul B. Henson]

> From: Ted Unangst
> Sent: Wednesday, June 28, 2017 8:50 PM
> 
> i'm afraid i won't make a very good ipmi maintainer, but i think i applied the
> patch in the right spot.

Cool, thanks; much appreciated.

Re: ipmi driver broken

[Paul B. Henson]

> From: Theo de Raadt
> Sent: Wednesday, June 28, 2017 8:41 PM
> 
> If you want it working, you will need to get it fixed.  On all
> machines, so that we can renable it.

I definitely don't want to be one of those entitled people demanding work
from developers without providing anything that you trounce upon ;). But
that's a bit of a big ask, make it work on all machines? I've got four
different models of supermicro servers that I certainly can do testing on,
although as I said, on these particular servers as far as I can tell (other
than the watchdog) the driver seems to work fine.

> Let me explain how we work.

I understand; really, I'm not asking you guys to invest a significant amount
of effort in improving the driver, or even technically "fixing" any new
issues or problems with it. I was only kindly requesting that you put back a
line that appears to have accidentally been deleted a few revisions ago that
broke it. So unless you're intentionally sabotaging it in preparation for
the ritual sacrifice :)?

It's too bad nobody else finds value in it; it provides sensors that aren't
otherwise available, provides access to the system event log for event data,
allows access to the management interface without needing to go through the
network, and ideally would allow access to the hardware watchdog.
Unfortunately I don't have expertise in low level hardware device driver
development so while I could be a tester I can't be a primary maintainer. So
if you guys end up scrapping it, I will be sad but that's the way it is. But
until then, given it works for me, it doesn't hurt to use it :). Or to ask
for one line to be put back so it would work in the shipped kernel; unless I
suppose said request results in it getting scrapped ;).

Thanks.

ipmi driver broken

[Paul B. Henson]

I noticed back when I upgraded to 5.9 the ipmi driver stopped working,
it just said:

ipmi0: get header fails
ipmi0: no SDRs IPMI disabled

I found the following post at the time which appeared to point out the
issue and suggest a fix:

http://openbsd-archive.7691.n7.nabble.com/fix-for-quot-ipmi0-get-header-fails-quot-td299427.html

After applying this and installing the resulting kernel, ipmi worked
fine. I skipped 6.0, but just updated my boxes to 6.1, and see the same
ipmi failures. It looks like this fix hasn't been applied, the code in
head is still missing this line. I applied it again to my 6.1 kernel and
it still seems to make ipmi work fine as far as I can tell.

Is there anyone maintaining ipmi or someone with commit privs that might
be kind enough to apply this so the next release version would have
working ipmi?

Thanks much...

Re: WARNING: symbol(icudt58_dat) size mismatch, relink your program

[Paul B. Henson]

On Sat, Aug 05, 2017 at 12:35:24AM +, Stuart Henderson wrote:

> The ports@ list is a better venue for ports-related queries,
> please see this: https://marc.info/?l=openbsd-ports=150157643516239=2

Ah, ok, thanks for the pointer.

> This is not preventing programs from running.

Hmm, I could've sworn I got that message and then slapd failed to start.
Dunno, maybe I got confused. Once I'm done working with openldap mdb I'll
start over from scratch and try again and see what happens.

Thanks for the info...

Re: umb device, SIM has no PIN?

[Paul B. Henson]

On Fri, Nov 24, 2017 at 11:08:25AM +, Stuart Henderson wrote:

> > booted under openbsd. The umb driver doesn't support accessing the card
> > directly for debugging and diagnostics?
> 
> Correct, you can't get at those from OpenBSD atm.

That's a bummer; guess you wouldn't care too much if things were working
:), but when you're trying to sort out why they're not it sure would be
nice. Then if you're placing an external antenna the signal strength
readings are cool.

> I don't have it handy to check now, but IIRC that's similar to what I
> see on MC8805 after adding the ID for fcc auth.

Interestingly, I tried the card in an external miniPCI to USB adapter,
and it worked fine? Without any driver changes or adding the fcc auth
ID. But when the card is installed directly in the system it doesn't
work :(. But it works fine under Linux, so it's not that the system
hardware is broken or incompatible.

It's a PC Engines APU 3, maybe the OpenBSD USB drivers for this board
aren't working quite right? With the external adapter, the driver sends
the MBIM_OPEN_MSG, gets an interrupt, receives a response, parses the
response, and moves on. Installed on the board, the driver sends the
MBIM_OPEN_MSG, and then nothing happens. No interrupt, no response,
nothing.

Jul 23 18:00:35 maggie /bsd: uhub1 at usb1 configuration 1 interface 0 "AMD EHCI
 root hub" rev 2.00/1.00 addr 1
Jul 23 18:00:35 maggie /bsd: uhub2 at uhub1 port 1 configuration 1 interface 0 
" Advanced Micro Devices product 0x7900" rev 2.00/0.18 addr 2
Jul 23 18:00:35 maggie /bsd: umb0 at uhub2 port 3 configuration 1 interface 12 
" Sierra Wireless, Incorporated Sierra Wireless MC7455 Qualcomm\M-.  
Snapdragon? X7 LTE-A" rev 2.10/0.06 addr 3

It looks like the card is on a stacked hub or something when installed
in the box? When it's plugged in on the external adapter:

Jul 23 18:15:38 maggie /bsd: umb1 at uhub0 port 4 configuration 1 interface 12 
" Sierra Wireless, Incorporated Sierra Wireless MC7455 Qualcomm\M-.  
Snapdragon? X7 LTE-A" rev 2.10/0.06 addr 2

It's connected directly to a controller. They seem to init the same:

Jul 23 18:00:35 maggie /bsd: umb0: umb_attach
Jul 23 18:00:35 maggie /bsd: umb0: ctrl_len=4096, maxpktlen=1422, cap=0x20
Jul 23 18:00:35 maggie /bsd: umb0: ctrl-ifno#12: ep-ctrl=5, data-ifno#13: 
ep-rx= 4, ep-tx=3
Jul 23 18:00:35 maggie /bsd: umb0: rx/tx size 16384/16384
Jul 23 18:00:35 maggie /bsd: umb0: umb_open
Jul 23 18:00:35 maggie /bsd: umb0: umb_ctrl_msg
Jul 23 18:00:35 maggie /bsd: umb0: -> snd MBIM_OPEN_MSG (tid 1)
Jul 23 18:00:35 maggie /bsd: umb0: sent MBIM_OPEN_MSG (tid 1)
Jul 23 18:00:35 maggie /bsd:0:   01 00 00 00 10 00 00 00 01 00 00 00 00 10 
0 0 00

Jul 23 18:21:20 maggie /bsd: umb1: umb_attach
Jul 23 18:21:20 maggie /bsd: umb1: ctrl_len=4096, maxpktlen=1422, cap=0x20
Jul 23 18:21:20 maggie /bsd: umb1: ctrl-ifno#12: ep-ctrl=5, data-ifno#13: 
ep-rx= 4, ep-tx=3
Jul 23 18:21:20 maggie /bsd: umb1: rx/tx size 16384/16384
Jul 23 18:21:20 maggie /bsd: umb1: umb_open
Jul 23 18:21:20 maggie /bsd: umb1: umb_ctrl_msg
Jul 23 18:21:20 maggie /bsd: umb1: -> snd MBIM_OPEN_MSG (tid 1)
Jul 23 18:21:20 maggie /bsd: umb1: sent MBIM_OPEN_MSG (tid 1)
Jul 23 18:21:20 maggie /bsd:0:   01 00 00 00 10 00 00 00 01 00 00 00 00 10 
0 0 00

But it's only when connected externally that the card actually generates
an interrupt and sends a response:

Jul 23 18:15:48 maggie /bsd: umb1: umb_intr
Jul 23 18:15:48 maggie /bsd: umb1: umb_intr: response available
Jul 23 18:15:48 maggie /bsd: umb1: umb_get_response_task
Jul 23 18:15:48 maggie /bsd: umb1: umb_decode_response
Jul 23 18:15:48 maggie /bsd: umb1: got response: len 16


Any thoughts on how to diagnose what might be a USB driver issue as
opposed to an LTE card issue 8-/?

Thanks...

umb device, SIM has no PIN?

[Paul B. Henson]

I'm trying to get an LTE card working in MBIM mode with the umb device
driver, but it just keeps saying "SIM not initialized PIN required". The
SIM isn't PIN locked, as far as I know the SIM has no PIN. I've tested
the card and SIM under linux on the exact same system and was able to
get it working fine just by supplying the APN.

The card is a Sierra Wireless MC7455; to get it working with the umb
driver I did have to disable the umsm driver as for some reason that one
claimed it first. Once that driver was disabled the umb driver seemed
happy with it:

umb0 at uhub2 port 3 configuration 1 interface 12 "Sierra Wireless, 
Incorporated Sierra Wireless MC7455 Qualcomm\M-. Snapdragon? X7 LTE-A" rev 
2.10/0.06 add r 3
ugen0 at uhub2 port 3 configuration 1 "Sierra Wireless, Incorporated Sierra 
Wireless MC7455 Qualcomm\M-. Snapdragon? X7 LTE-A" rev 2.10/0.06 addr 3

After boot, the interface looked like:

umb0: flags=8810 mtu 1500
index 6 priority 0 llprio 3
roaming disabled registration unknown
state down cell-class none
SIM not initialized PIN required
status: down

I set the APN and tried to bring it up:

umb0: flags=8811 mtu 1500
index 6 priority 0 llprio 3
roaming disabled registration unknown
state down cell-class none
SIM not initialized PIN required
APN r.ispsn
status: down

But it still just says the SIM is not initialized. After a minute or two,
it starts logging these to the console:

umb0: state change timeout
umb0: state change timeout
umb0: state change timeout
umb0: state change timeout


Am I missing something? This card isn't listed explicitly as being
compatible, is there a problem with the driver and this particular card?

Under linux, the serial control interfaces were available as USB devices
so you could poke at the card with AT commands, I don't see any listed
booted under openbsd. The umb driver doesn't support accessing the card
directly for debugging and diagnostics?

Thanks...

Re: kernel reordering and config -e

[Paul B. Henson]

On Wed, Nov 22, 2017 at 04:45:59PM +, Kevin Chadwick wrote:

> I believe the second scenario would need /dev/mem access making it a
> larger change than it first appears (config with a new option could
> possibly save the original kernel file and compare the two kernel
> files).

Ah, I didn't mean that; I meant save your interactive 'config -e'
session in a file that could be played back later. IE, you run 'config
-e - /etc/ukc.conf ...', then type 'change x', 'disable y' etc,
and then when you 'quit', config would write a transcript of your
changes to /etc/ukc.conf such that 'config -e -

broken EHCI USB on AMD chipset?

[Paul B. Henson]

I have a pcengines APU 3 system, which has both USB3 and USB2 ports:

ehci0 at pci0 dev 18 function 0 "AMD Hudson-2 USB2" rev 0x39: apic 4 int 18
ehci1 at pci0 dev 19 function 0 "AMD Hudson-2 USB2" rev 0x39: apic 4 int 18

xhci0 at pci0 dev 16 function 0 "AMD Bolton xHCI" rev 0x11: msi

The USB2 ports seem to be broken. I initially was having trouble getting
an LTE modem to work, but then noticed more general underlying issues.
If a USB device is connected when the system boots, it will find it;
however, if you hot plug a USB device after the system is up, it doesn't
notice. Further, if you unplug a device while the system is up, it
doesn't notice it was removed. It appears that the system isn't
receiving interrupts for the EHCI USB devices, the interrupt count from
vmstat -i :

irq101/ehci0   190
irq101/ehci1   460

does not change when I plug in or remove a device, and I think the
reason the LTE modem is not working is because the cell modem driver never
receives a response from the commands sent to the modem, presumably because
the interrupt notifying the USB driver the data is ready to read is
never seen/handled.

The xhci usb3 ports work fine, they hot plug/remove devices correctly,
the lte modem works when plugged into them, and the vmstat interrupt
count for irq99/xhci0 increases when devices are using those ports.

The EHCI ports seem to work fine under Linux, including the LTE modem
when attached to them, so this seems to be an issue with openbsd, not
faulty hardware per se. The Linux driver does have a couple of
workarounds in their EHCI driver for AMD chipsets, I'm not sure if
either of them are relevant for this; one involves disabling low power
mode during transfers and the other says:

"EHCI controller on AMD SB700/SB800/Hudson-2/3 platforms may
read/write memory space which does not belong to it when
there is NULL pointer with T-bit set to 1 in the frame list
table. To avoid the issue, the frame list link pointer
should always contain a valid pointer to a inactive qh"

I don't see anything specifically discussing flaky interrupts. Any
thoughts on what might be going on here with USB and how it fix it?

Thanks...

pcengines apu boards

[Paul B. Henson]

I was wondering if anybody is successfully running openbsd on pcengines apu
boards? I have one of their APU3 series, specifically a apu3b4 with OpenBSD
6.2 on it but I can't get the USB2 EHCI ports functioning correctly (for one
thing, they don't detect a hot plugged device), I'm not sure if it's an
issue with the ehci driver and the amd ehci chipset or possibly something in
the bios acpi tables. But just as a data point, it would be interesting to
know if the problem is specific to my board or endemic to the design, so if
anyone has an APU series board with fully functional USB2 ports on the ehci
controller, I would much appreciate hearing which board it is, which
specific AMD chipset is driving the controller, and what bios version you
are running (and what OpenBSD version too).

Thanks much.

Re: pcengines apu boards

[Paul B. Henson]

> From: Bryan Everly
> Sent: Thursday, November 30, 2017 2:46 PM
> 
> I'm running my primary firewall at home on an apu2...

Cool. Have you ever tried using an internal Mini PCI card in it?

Re: pcengines apu boards

[Paul B. Henson]

> From: Base Pr1me
> Sent: Thursday, November 30, 2017 2:08 PM
> 
> I run 5 apu2 devices with no problems. I don't have any apu3 devices ... yet.

Thanks for the feedback. Do you by any chance have any USB type Mini PCI cards 
installed internally? I initially noticed the issue with a mini PCI LTE modem 
card. Then I realized it was a more generic USB problem; I believe the apu2 has 
USB1 and USB2 ports, the apu3 has two USB3 ports externally, and then the mini 
PCI and a couple of internal headers are USB2. The USB3 ports, using the xHCI 
driver, work fine, I suppose in the worst case I could use an external Mini PCI 
to USB adapter and plug the card in outside of the case, but that just seems so 
kludgy .

I actually found a friend locally who had a apu2 board, he couldn't get the LTE 
card to work on the internal mini PCI slot, which also appeared to be EHCI 
based, and it would sometimes work and sometimes not plugged into the external 
USB ports. It was really weird, when plugged into the same external port, 
sometimes the device would show up on the EHCI bus (and not work) and sometimes 
it would show up on the OHCI bus (and work). He didn't seem to have any trouble 
with USB flash drives on the EHCI bus on his apu2 though.

Re: pcengines apu boards

[Paul B. Henson]

> From: Eike Lantzsch
> Sent: Thursday, November 30, 2017 3:12 PM
> 
> here: APU2C4 with one SATA drive of 6TB and one 4TB via USB3 and an

Hmm, I didn't think the apu2 had USB3, but double checking the specs I see
it does. My friend that said he had an APU2 must actually have an original
APU, as his board doesn't have USB3. Yeah, the external xHCI USB3 ports work
fine on my APU3, it's the EHCI ones that are screwed up, they are only
available via two internal headers or if you use the Mini PCI slot. There
probably aren't very many people that are routing the internal USB headers
to external connectors, so unless somebody is using a USB Mini PCI expansion
card on an APU2/3, they probably aren't using the EHCI controller.

Thanks for the info.

Re: broken EHCI USB on AMD chipset?

[Paul B. Henson]

On Tue, Nov 28, 2017 at 08:03:05PM -0800, Paul B. Henson wrote:

> The EHCI ports seem to work fine under Linux, including the LTE modem
> when attached to them, so this seems to be an issue with openbsd, not
> faulty hardware per se.

I tested FreeBSD on this box as well, it detected the EHCI ports as:

usbus1: EHCI version 1.0
usbus1 on ehci0
usbus1: 480Mbps High Speed USB v2.0
usbus2: EHCI version 1.0
usbus2 on ehci1
usbus2: 480Mbps High Speed USB v2.0
ugen1.1:  at usbus1
ugen2.1:  at usbus2
uhub0:  on usbus1
 on usbus2
ugen1.2:  at usbus1
uhub3: 
on usb
us1
ugen2.2:  at usbus2
uhub4: 
on usb
us2

As far as I can tell the ports work ok under FreeBSD, detecting hot plug
and removal of devices, and the interrupt count from vmstat -i increases
when doing so. FreeBSD doesn't support the Sierra Wireless card I have
but I'm guessing it would work.

So it just seems to be an issue with OpenBSD and this board or USB
chipset or something. I turned on debugging in the ehci and uhub code,
but when I plug something in nothing whatsoever happens, so that wasn't
very useful. Any suggestions on other debugging to enable or any other
approach to figure out what's going on here?

Thanks...

Re: umb device, SIM has no PIN?

[Paul B. Henson]

> The card is a Sierra Wireless MC7455; to get it working with the umb

Looking at the source code, I see that there's an workaround for the
EM7455 card, something about requiring an "FCC Authentication" command?
>From what I understand the MC7455 is the same as the EM7455 other than
form factor, so I added it to the list for that workaround and also
turned on debugging in the driver. Here's what it has to say now:

Jul 23 18:12:41 maggie /bsd: umb0 at uhub2 port 3 configuration 1
interface 12 "Sierra Wireless, Inc
orporated Sierra Wireless MC7455 Qualcomm\M-. Snapdragon? X7 LTE-A" rev
2.10/0.06 addr 3
Jul 23 18:12:41 maggie /bsd: umb0: ctrl_len=4096, maxpktlen=1422,
cap=0x20
Jul 23 18:12:41 maggie /bsd: umb0: ctrl-ifno#12: ep-ctrl=5,
data-ifno#13: ep-rx=4, ep-tx=3
Jul 23 18:12:41 maggie /bsd: umb0: rx/tx size 16384/16384
Jul 23 18:12:41 maggie /bsd: umb0: -> snd MBIM_OPEN_MSG (tid 1)
Jul 23 18:12:41 maggie /bsd: umb0: sent MBIM_OPEN_MSG (tid 1)
Jul 23 18:12:41 maggie /bsd:0:   01 00 00 00 10 00 00 00 01 00 00 00
00 10 00 00
Jul 23 18:12:41 maggie /bsd: umb0: vers 1.0
Jul 23 18:12:41 maggie /bsd: ugen0 at uhub2 port 3 configuration 1
"Sierra Wireless, Incorporated Si
erra Wireless MC7455 Qualcomm\M-. Snapdragon? X7 LTE-A" rev 2.10/0.06
addr 3


Jul 23 18:13:31 maggie /bsd: umb0: stop: reached state DOWN  
Jul 23 18:13:59 maggie /bsd: umb0: init: opening ...
Jul 23 18:13:59 maggie /bsd: umb0: -> snd MBIM_OPEN_MSG (tid 2)
Jul 23 18:13:59 maggie /bsd: umb0: sent MBIM_OPEN_MSG (tid 2)
Jul 23 18:13:59 maggie /bsd:0:   01 00 00 00 10 00 00 00 02 00 00 00
00 10 00 00
Jul 23 18:14:29 maggie /bsd: umb0: state change timeout
Jul 23 18:14:29 maggie /bsd: umb0: init: opening ...
Jul 23 18:14:29 maggie /bsd: umb0: -> snd MBIM_OPEN_MSG (tid 3)
Jul 23 18:14:29 maggie /bsd: umb0: sent MBIM_OPEN_MSG (tid 3)
Jul 23 18:14:29 maggie /bsd:0:   01 00 00 00 10 00 00 00 03 00 00 00
00 10 00 00
Jul 23 18:14:59 maggie /bsd: umb0: state change timeout

Not sure where to go from here.

Re: pcengines apu boards

[Paul B. Henson]

> From: Marko Cupac
> Sent: Monday, December 4, 2017 3:54 AM
> 
> I have just ordered one APU3b4, as I wanted to test mobile provider as
> a backup link. I see it probably won't be any good as OpenBSD router
> (yet), but at least I'll be able to test and give feedback.

Assuming you're planning to use an internal Mini PCI card, unless you have more 
luck than me, it's not going to work :(. I'm hoping I will be able to fix the 
EHCI driver to be more happy with the AMD USB chipset, but this point I'm still 
fumbling with it :).

rdomain/rtable

[Paul B. Henson]

I've got a box with an LTE cellular modem in it whose purpose is to provide
a backup connection to the Internet if the hardwire service goes down. It's
running OSPF to connect to the rest of the network, and the only time any
traffic should go over the cellular link (which is slower and bandwidth
capped) is if the hardwire interconnection is down, including ideally
traffic generated from the system itself.

I have that part working, by adding in a local static default route to the
cellular gateway with less priority than the OSPF default route. However,
for testing purposes, I'd like to be able to poke out the cellular link on
an as-needed basis without having to switch the entire box over to using it.
Virtual routing tables looked perfect for this purpose, as I could just
spawn a single process with a different default route, we do something
similar with network name spaces under Linux.

However, I can't quite get it to work. What I'd really like is to be able to
make a copy of the current system routing table, then change one thing about
it. However, a new rdomain shows up with no routes or interfaces in the
routing table. I can add the new default route pointing out the cellular
link, and get traffic to go out there. But I haven't sorted out how to make
all the traffic for my internal network still go through the internal link
rather than get sent out the default route. While ideally all the OSPF
routes would propagate to the other routing domain I tried just adding a
static to the /16 for our internal address space:

Internet:
DestinationGatewayFlags   Refs  Use   Mtu  Prio
Iface
default24.x.x.x  UGS06 - 8 umb0
10.0/1610.128.0.21UGS00 - 8 em0

That doesn't work; the documentation says you need to get pf to pass packets
across routing domains. However, it says:

rtable number
Used to select an alternate routing table for the routing lookup.
Only effective before the route lookup happened, i.e. when
filtering inbound.

Unfortunately, for traffic originating from the system itself, there isn't
really an "inbound" interface? So I'm not sure what pf rule would make this
work. Is it just not possible, or am I missing something?

Thanks much.

Re: kernel reordering and config -e

[Paul B. Henson]

On Mon, Nov 20, 2017 at 08:37:43AM +, Roderick wrote:

> Commenting out the line "/usr/libexec/reorder_kernel &" at the
> end of rc?
> 
> I suspect it is not forseen not to benefice of KARL.

No, actually, if the hash of the kernel is different than expected, the
reorder_kernel aborts and doesn't generate a new one. So you don't need
to do anything explicitly after the config -e to avoid your change being
wiped out. What I did was update the saved hash with one matching my
modified kernel (not quite understanding what was going on yet) which
caused KARL to wipe my changes out with the default.

Sierra Wireless MC7455 LTE cell network card

[Paul B. Henson]

I'm trying to get the subject card to work under OpenBSD 6.2; it works
fine under Linux so I know the card itself and its SIM etc are correctly
configured and functional.

The card is set to MBIM mode, and I'd like to use the umb driver rather
than the umsm driver as not to have to muck with PPP. It seems this card
is detected first by the umsm driver though, as I had to disable that
driver for the card to be picked up by umb. The umb man page says
"Devices which fail to provide a conforming MBIM implementation will
probably be attached as some other driver", does this indicate the
MC7455 (as opposed to the EM7455, which is explicitly listed as
compatible) isn't recognized as an MBIM device? It seems to work fine in
MBIM mode under linux, and the umb driver does find it once umsm is
disabled.

Is there any way to access the serial interface of the device under
openbsd in order to execute diagostic AT commands? Under linux in
addition to the network device the card also generates a few USB serial
devices, one of which can be used to run commands on it. I saw such
devices with the umsm driver, but I don't see any with the umb driver. I
haven't gotten any farther than installing the card and getting the umb
driver to recognize it at this point, but it would be nice to be able to
poke at it and see what the card has to say for itself.

Thanks...

Re: kernel reordering and config -e

[Paul B. Henson]

On Mon, Nov 20, 2017 at 06:50:30AM +0100, Sebastien Marie wrote:

> When it did that, it uses the object (I didn't recall the exact name)
> with the previous mentioned array, with *default* configuration. So the
> previous modification done with config(8) is cleared.

Yeah, I figured that out after I updated the saved KARL hash and then my
box came up with no serial console :).

> For me, there is currently no way to ask config(8) to alter the right
> file in /usr/share/relink/kernel to "ship" the modification in all
> future generated KARL kernels.

I thought that might be the case; maybe someday config(8) will be
extended to work with the object files as well as the kernel binary
itself to allow that.

> - makes your changes in /usr/src/sys, build and install a new no-GENERIC
>   kernel (and do it at each upgrade)

If I do that, can the resultant object files (which will have my com2
irq change) be used with KARL? Hmm, it seems like all I really need to
do is compile a new com_isa.o and drop it in to the existing directory?
Or replace whichever object file contains the constant I need to change;
it's not like I'm modifying code or making any drastic changes... Hmm,
I'll have to compile a new kernel and poke at it; it'll just be a matter
of remembering to redo it after patches, but I already had to redo the
config -e anyway.

Thanks...

kernel reordering and config -e

[Paul B. Henson]

I just updated a server to 6.2; unfortunately this box has an oddball
SOL com2 on irq10 so I need to run 'config -e' on the kernel to update
it and make the serial console work. I noticed afterwards in the boot
messages it was complaining about kernel reordering failures, and
thinking I was fixing it, I updated the file /var/db/kernel.SHA256 with
the hash of my modified kernel. I quickly discovered that resulted in a
successfully reordered kernel with a stock com2 irq :(.

I didn't see anything in the config man page or faq about interaction
between kernel reordering and config on a binary kernel. In hindsight I
see that the hash check is to keep from replacing a locally modified
kernel.  Is there a supported way to both fix hardcoded settings on a
stock kernel and use reordering? Or do you need to update your settings
in the config and compile a kernel from scratch? If you do, does
/usr/share/compile automatically get populated with your new kernel
objects and reordering just starts working, or do you need to do
something manually to get it running with a locally compiled kernel?

Thanks...

Re: kernel reordering and config -e

[Paul B. Henson]

On Mon, Nov 20, 2017 at 02:01:56PM -0700, Theo de Raadt wrote:

> If someone wants to solve this fully there have been some proposals
> for keeping track of the instruction sequence, and attempting to
> reapply it upon each relink in the build directory.  There just hasn't
> been any scripting changes to do that from anyone, and it isn't on my
> radar as important.

Ah, rather than make binary changes to the object files that get linked,
just redo the changes to the resultant kernel binary every time it is
generated. That's definitely simpler to do with the existing tools.

I see someone made a suggestion that you replied to with a classic
"where's the patch" :), I don't think he was suggesting someone else do
it but more looking for guidance on what you'd considerable acceptable
before spending time on it.

For example, would the basic "user manually constructs a text file with
config commands by hand that then just gets passed to config on stdin"
approach he mentioned be good enough to commit? Or would you want
something more integrated into config where it would have a new command
that would generate a file based on the current session, and a new
option to process changes from a file rather than interactively? It
looks like it would be difficult to detect errors in the first scenario,
and I don't know if that would be an issue.

Thanks...

Re: kernel reordering and config -e

[Paul B. Henson]

On Tue, Nov 21, 2017 at 09:49:37AM +, Dimitris Papastamos wrote:

> This is what I do in rc.shutdown to handle this case:
> 
> /usr/bin/printf "disable inteldrm*\nquit\n" | /usr/sbin/config -ef /bsd
> /bin/sha256 -h /var/db/kernel.SHA256 /bsd

Cool, thanks for the suggestion; that should be good as long as the box
doesn't panic or otherwise have an unclean shutdown.

Re: help updating EHCI driver

[Paul B. Henson]

> From: Martin Pieuchot
> Sent: Thursday, December 7, 2017 3:18 AM
> 
> Which issue are you having?

Sorry, there was more context in an earlier thread. Basically, I have a pc 
engines APU3 board which has AMD Hudson-2 EHCI USB ports on it. If devices are 
plugged in when the system boots and the ports are initialized, the operating 
system sees they are there. However, if you hot plug a device after it is 
booted, or remove a device that was plugged in at boot, the system does not 
notice the change in state. Also, the Sierra wireless LTE modem I was trying to 
use does not function, the driver sends an open message over the USB bus to the 
device and then nothing ever comes back. Once the system is booted, the 
interrupt account for the ECHI ports from 'vmstat -I' never increases. The xHCI 
USB ports on the same system seemed to work fine, detecting hot plug/remove 
events, and properly initializing the wireless modem.

> What makes you think that the quirks below
> will help?  What do you mean with 'work fine on those systems'?  If they
> work fine, which issues are you having?

The same board when booted up under either Linux or FreeBSD appears to have 
fully functional EHCI USB ports; they detect hot plug/remove events, and under 
linux the wireless modem is initialized and can successfully pass traffic 
(FreeBSD doesn't have a driver for it, so I was unable to test it there).

I honestly don't know if these specific quirks will resolve the issue under 
OpenBSD, all I know is that there is something Linux/FreeBSD is doing different 
regarding the USB hardware, and porting these quirks seemed a good place to 
start. I'm not really a low level hardware/device driver guy, so I'm flying a 
bit blind. Someone told me there are known issues with amd USB ports in 
general, such as ath based USB wireless cards not working, so these might help 
that problem even if it doesn't fix mine.

> It depends how the controller is connected to the host.  If you look at
> the PCI glue driver, dev/pci/ehci_pci.c you'll see
> 
> 115:  /* Map I/O registers */
> 116:  if (pci_mapreg_map(pa, PCI_CBMEM, PCI_MAPREG_TYPE_MEM, 0,
> 117: >sc.iot, >sc.ioh, NULL, >sc.sc_size, 0))
> 
> Then in the EHCI driver, dev/usb/ehci.c, these are accessed via the
> EREAD/EWRITE/EOREAD/EOWRITE macros.

Ah, ok; it appears that pci_mapreg_map defines the start of the region as:
 
ex = pa->pa_ioex;
if (ex != NULL) {
start = max(PCI_IO_START, ex->ex_start);

with:

#define PCI_IO_START  0

So the starting memory address is either 0 or pa->pa_ioex from the struct 
pci_attach_args that was passed into ehci_pci_attach.

Given the existing reads:

sc->sc.sc_offs = EREAD1(>sc, EHCI_CAPLENGTH);
define EHCI_CAPLENGTH 0x00

I'm pretty sure it's not zero, so it must be the one from  pa_ioex.

> But maybe you just want to use pci_conf_read()/pci_conf_write()?

Hmm, given my lack of detailed knowledge of this area I can't say for sure. 
However, there are two different things being done in the linux code I am 
referring to:

outb_p(0xe0, 0xcd6);

This I believe writes the byte 0xe0 to the  I/O port at address 0xcd6, where as 
this:

pci_write_config_dword(amd_chipset.nb_dev, 0xe4, val);

Writes the contents of the variable val to the PCI configuration register 
located at 0xe4. Those are two different operations, right?

So wherever the linux code writes to an absolute I/O port for the USB device, 
such as 0xcd6, I can subtract the beginning of the mapped region as stored in 
pa_ioex from it to arrive at the appropriate offset value to use with 
EREAD/WRITE?

> Is low power mode enabled on OpenBSD?

Based on the comment in the linux code:

"The hardware normally enables the A-link power management feature, which
lets the system lower the power consumption in idle states."

I believe that is the default behavior of the hardware unless you explicitly do 
something otherwise with it.

 > The other quirk involves never having an empty frame list; I have
> > implemented the logic to detect when that is required, but haven't even
> > come close to wrapping my head around actually implementing the quirk
> > itself.
> 
> For which transfer type is this quirk required, isochronous only?

The explanation of this quirk is:

"EHCI controller on AMD SB700/SB800/Hudson-2/3 platforms may
read/write memory space which does not belong to it when
there is NULL pointer with T-bit set to 1 in the frame list
table. To avoid the issue, the frame list link pointer
should always contain a valid pointer to a inactive qh."

I am also unfortunately not that expert in the underlying USB hardware level 
protocol, but the quirk is referenced in two functions, scan_isoc:

if (!ehci->use_dummy_qh ||
q.itd->hw_next != EHCI_LIST_END(ehci))
*hw_p = q.itd->hw_next;
else
 

Re: pcengines apu boards

[Paul B. Henson]

On Sat, Dec 02, 2017 at 10:40:14PM +1000, Douglas Ray wrote:

> On the APU3a4 the internal USB headers were broken.
> I had email from pcengines (March 2017) saying this would
> be addressed in the APU3b series., but we went for APU2.

I have a APU3b series, they fixed the incorrect pinout on the internal
usb headers. The internal ECHI ports work fine under both linux and
freebsd connected to a USB backplate I'm testing with. It's definitely a
disagreement between the AMD EHCI USB chipset and OpenBSD . I'm
going to see if I can port some of the workarounds and quirks for that
chipset from linux/freebsd to the openbsd driver and see if I have any
luck getting it working; drivers aren't my strong suite but we'll see
what happens. In the worst case I guess I'll use an external miniPCI to
USB adapter and connect my LTE modem to the external xHCI ports, they
seem to work fine under OpenBSD.

Thanks...

help updating EHCI driver

[Paul B. Henson]

I'm trying to port some quirks for AMD USB chipsets from other operating
systems to OpenBSD to hopefully resolve issues I am having with the pc
engines APU3 EHCI ports, as they seem to work fine on those systems.
I've got a pretty rough draft of one of them, which disables low-power
mode during transfers, but would appreciate a little clarification on
device I/O as I'm not generally a device driver developer.

Under Linux, the kernel uses absolute addresses when it's doing port I/O
to a device, so that's what I am referencing in their implementation. In
OpenBSD I see that a driver maps a handle to a region of memory and then
uses offsets from the base of that region for port I/O. It looks like
the EHCI driver code has already mapped that region and the handle is
available for me to use, but I don't see where that mapping was made or
how to figure out what the base was in order to turn the absolute
addresses I have into appropriate offsets to use with the openbsd API?

Then, for some of the chipsets, in addition to poking at the USB device
itself to twiddle the low-power mode, you also have to muck with the
northbridge configuration. I think I gathered the device information,
although I don't know that was the correct way to do so; but I need to
map the I/O region for it to a handle so I can modify it. If a driver for
one device needs to write to a different device is it supposed to call
bus_space_map on its own to get a mapping, or can it somehow get access
to the existing one already in place for that device?

Finally, low power mode is supposed to be disabled whenever there are
asynchronous transfers occurring, and then re-enabled once they
complete. I'm not sure I've put the calls in the right place, and I know
I haven't handled the case where transfers fail or are canceled  rather
than complete.

The other quirk involves never having an empty frame list; I have
implemented the logic to detect when that is required, but haven't even
come close to wrapping my head around actually implementing the quirk
itself.

In any case, here is my current laughable diff, advice and corrections
most appreciated.


Index: pci/ehci_pci.c
===
RCS file: /cvs/src/sys/dev/pci/ehci_pci.c,v
retrieving revision 1.30
diff -u -p -r1.30 ehci_pci.c
--- pci/ehci_pci.c  20 Jul 2016 09:48:06 -  1.30
+++ pci/ehci_pci.c  6 Dec 2017 02:46:24 -
@@ -66,6 +66,8 @@ struct ehci_pci_softc {
 };
 
 int ehci_sb700_match(struct pci_attach_args *pa);
+int ehci_amd_pll_quirk_match(struct pci_attach_args *pa);
+int ehci_amd_pll_quirk_match_nb(struct pci_attach_args *pa);
 
 #define EHCI_SBx00_WORKAROUND_REG  0x50
 #define EHCI_SBx00_WORKAROUND_ENABLE   (1 << 3)
@@ -111,6 +113,7 @@ ehci_pci_attach(struct device *parent, s
char *devname = sc->sc.sc_bus.bdev.dv_xname;
usbd_status r;
int s;
+   struct pci_attach_args amd_pa;
 
/* Map I/O registers */
if (pci_mapreg_map(pa, PCI_CBMEM, PCI_MAPREG_TYPE_MEM, 0,
@@ -131,6 +134,86 @@ ehci_pci_attach(struct device *parent, s
 
/* Handle quirks */
switch (PCI_VENDOR(pa->pa_id)) {
+   case PCI_VENDOR_AMD:
+   /* AMD errata indicates 8111 chipset EHCI is broken */
+   if (PCI_PRODUCT(pa->pa_id) == PCI_PRODUCT_AMD_8111_EHCI) {
+   printf("%s: AMD 8111 EHCI broken, skipping", devname);
+   goto disestablish_ret;
+   }
+   if (pci_find_device(_pa, ehci_amd_pll_quirk_match)) {
+   sc->sc.amd_chipset.rev = PCI_REVISION(amd_pa.pa_class);
+   if (PCI_PRODUCT(amd_pa.pa_id) == 
PCI_PRODUCT_ATI_SBX00_SMB) {
+   if (sc->sc.amd_chipset.rev >= 0x10 &&
+   sc->sc.amd_chipset.rev <= 0x1f)
+   sc->sc.amd_chipset.gen = 
AMD_CHIPSET_SB600;
+   else if (sc->sc.amd_chipset.rev >= 0x30 &&
+sc->sc.amd_chipset.rev <= 0x3f)
+   sc->sc.amd_chipset.gen = 
AMD_CHIPSET_SB700;
+   else if (sc->sc.amd_chipset.rev >= 0x40 &&
+sc->sc.amd_chipset.rev <= 0x4f)
+   sc->sc.amd_chipset.gen = 
AMD_CHIPSET_SB800;
+   else
+   sc->sc.amd_chipset.gen = 
AMD_CHIPSET_UNKNOWN;
+
+   }
+   else if (PCI_PRODUCT(amd_pa.pa_id) == 
PCI_PRODUCT_AMD_HUDSON2_SMB) {
+   if (sc->sc.amd_chipset.rev >= 0x11 &&
+   sc->sc.amd_chipset.rev <= 0x14)
+   sc->sc.amd_chipset.gen = 
AMD_CHIPSET_HUDSON2;
+   else if (sc->sc.amd_chipset.rev >= 0x15 &&
+

Re: 3g modem support

[Paul B. Henson]

> From: Marko Cupac
> Sent: Wednesday, December 6, 2017 2:47 AM
> 
> ...which suggests some Sierra Wireless modems, none of which are
> available for purchase in the country I live in.

I've got the MC7455, which I believe is basically the same as the EM7455. 
Presumably this might be one of the cards you say you can't get though. I 
haven't been able to thoroughly test it given my issues with the APU3, but a 
friend of mine played with it a bit under vmware and it seems functional under 
both the umsm driver with PPP and the umb driver in MBIM mode, although in 
order to use the latter you need to disable the umsm module as it claims the 
device with a higher priority.

I ended up ordering one of these:

https://www.amazon.com/gp/product/B01JGCSPEA/ref=oh_aui_detailpage_o00_s00?ie=UTF8=1

and will most likely connect the miniPCI card to the external xHCI ports on the 
APU3 which at least at initial glance seems functional. Kind of kludgy, but 
better than the other fallback plan of using linux for this deployment :).