Re: Another potential awk or xargs bug?

2021-04-16 Thread Jordan Geoghegan



On 4/15/21 7:49 AM, Otto Moerbeek wrote:
> On Thu, Apr 15, 2021 at 04:29:17PM +0200, Christian Weisgerber wrote:
>
>> Jordan Geoghegan:
>>
>>> --- /tmp/bad.txt  Wed Apr 14 21:06:51 2021
>>> +++ /tmp/good.txt  Wed Apr 14 21:06:41 2021
>> I'll note that no characters have been lost between the two files.
>> Only the order is different.
>>
>>> The only thing that changed between these runs was me using either xargs -P 
>>> 1 or -P 2.
>> What do you expect?  You run two processes in parallel that write
>> to the same file.  Obviously their output will be interspersed in
>> unpredictable order.
>>
>> You seem to imagine that awk's output is line-buffered.  But when
>> it writes to a pipe or file, its output is block-buffered.  This
>> is default stdio behavior.  Output is written in block-size increments
>> (16 kB in practice) without regard to lines.  So, yes, you can end
>> up with a fragment from a line written by process #1, followed by
>> lines from process #2, followed by the remainder of the line from
>> #1, etc.
>>
>> -- 
>> Christian "naddy" Weisgerber  na...@mips.inka.de
>>
> Right, a fflush() call after the printf makes the issue go away, but
> only since awk is being nice and issues a single write call for that
> single printf. Since awk afaik does not give such a guarantee, it is
> better to have each parallel invocation write to a separate file and
> then cat them together after all the awk runs are done.
>
>   -Otto

Hello Christian and Otto,

Thank you for setting me straight. The block vs line buffering issue should 
have been obvious to me. What got me confused was that this solution worked 
well, for a long time - until it didn't. One would assume that it would 
consistently mangle output...

While fflush does seem to fix the issue, I wanted to explore your suggestion 
Otto of writing to a temporary file from within awk.

Is something like the following a sane approach to safely generating temporary 
files from within awk?:

BEGIN{ cmd = "mktemp -q /tmp/workdir/tmp.XXX" ; if( ( cmd | getline result 
) > 0 ) TMPFILE = result ; else exit 1 }

Unless I'm missing something obvious, It seems there is no way to capture both 
the stdout and return code of an external command from within awk. My 
workaround solution to error check the call to mktemp here is to abort if 
mktemp returns no data. Is this sane?

Regards,

Jordan



Another potential awk or xargs bug?

2021-04-15 Thread Jordan Geoghegan
Hello,

I've found some very interesting behaviour when subjecting various awk 
implementations to some very specific circumstances.

I'm basically looking for a sanity check here to confirm if I'm just wildly 
flailing, or if I am indeed onto something here.

Here's my situation:

When parsing some RIR data in parallel using awk with xargs, I seem to have 
found a way to reliable lose and/or mangle output with parallel xargs. My 
google-fu seems to be failing me. I understand that xargs does not buffer 
output and that lines may arrive out of order, but in this case I am reliably 
and reproducibly losing data and receiving mangled output. But wait, it gets 
stranger.

I don't want to lose you guys here with a long winded explanation, so I'm going 
to show you a diff that shows reproducibly mangled output when using xargs in 
parallel mode:

--- /tmp/bad.txt  Wed Apr 14 21:06:51 2021
+++ /tmp/good.txt  Wed Apr 14 21:06:41 2021
@@ -1,5 +1,3 @@
-267386
-A264890
 AS262399
 AS262400
 AS262401
@@ -1774,6 +1772,7 @@
 AS264887
 AS264888
 AS264889
+AS264890
 AS264891
 AS264892
 AS264893
@@ -3552,6 +3551,7 @@
 AS267383
 AS267384
 AS267385
+AS267386
 AS267387
 AS267388
 AS267389
@@ -4220,6 +4220,7 @@
 AS268318
 AS268319
 AS268320
+AS268320
 AS268321
 AS268321
 AS268323
@@ -7785,6 +7786,7 @@
 AS270633
 AS270633
 AS270634
+AS270634
 AS270635
 AS270635
 AS270636
@@ -10277,5 +10279,3 @@
 AS46210
 AS46280
 AS46280
-ASAS268320
-ASS270634

The only thing that changed between these runs was me using either xargs -P 1 
or -P 2.

To allow folks to follow along with me at home, I've included the two files 
(gzipped for politeness) I used to trigger this behaviour.

Once you've extracted the attached text files into your working directory, 
here's a snippet that should reproduce my issue:

$ printf 'BR\nCA\n' > cc.txt

$ find . -type f -name "[12].txt" -print0 | xargs -0 -n 1 -P 2 -- awk -F '|' 
'NR==FNR { A[$1]=1 ; next } $1 in A && $2 == "asn" { printf("AS%s\n", $3) }' 
cc.txt

What does this 1 liner do, well it's supposed to slurp the country codes 
specified in cc.txt into an array where we then check the first field of each 
row of the RIR data against. If the first field matches a country code in the 
array and the second field indicates that this row is an ASN record, then we 
print the 3rd field prepended with 'AS'. As you can see, if you grep the output 
of the above command for the string "ASAS", "ASS" or 'A2' you should see some 
mangled ASNs. If you change "-P 2" to "-P 1" this mangling will not occur.

Here's where things get very weird. While parsing this data (as part of a 
larger dataset comprising an aggregation of all the registrar delegation 
statistics) I've been using this snippet for a while to quickly fetch ASN 
records. It is not until I have BOTH the BR and CA country codes in the array 
that I can trigger this bug. I can have any number of country codes in the 
array, but if Brazil AND Canada happen to be specified in the array, then I get 
mangled output, but ONLY if executed with parallel xargs. This reproducibly 
happens when using awk, gawk or mawk. To further melt your brain, this 
behaviour has NOT been observed when using goawk, a POSIX compliant awk 
implementation written in go.

Just to prove my point, here's me testing the hash outputs between various awk 
implementations with my above 1 liner:

$ find . -type f -name "[12].txt" -print0 | xargs -0 -n 1 -P 2 -- awk -F '|' 
'NR==FNR { A[$1]=1 ; next } $1 in A && $2 == "asn" { printf("AS%s\n", $3) }' 
cc.txt | sort | md5
    2a20f44ce6a23d5c49b05b9f2689ef93

$ find . -type f -name "[12].txt" -print0 | xargs -0 -n 1 -P 1 -- awk -F '|' 
'NR==FNR { A[$1]=1 ; next } $1 in A && $2 == "asn" { printf("AS%s\n", $3) }' 
cc.txt | sort | md5
    9ab3dbfbff5746f059cdb35221ff73b1
---
$ find . -type f -name "[12].txt" -print0 | xargs -0 -n 1 -P 2 -- mawk -F '|' 
'NR==FNR { A[$1]=1 ; next } $1 in A && $2 == "asn" { printf("AS%s\n", $3) }' 
cc.txt | sort | md5
    2a20f44ce6a23d5c49b05b9f2689ef93

$ find . -type f -name "[12].txt" -print0 | xargs -0 -n 1 -P 1 -- mawk -F '|' 
'NR==FNR { A[$1]=1 ; next } $1 in A && $2 == "asn" { printf("AS%s\n", $3) }' 
cc.txt | sort | md5 >
    9ab3dbfbff5746f059cdb35221ff73b1
---
$ find . -type f -name "[12].txt" -print0 | xargs -0 -n 1 -P 2 -- 
~/go/bin/goawk -F '|' 'NR==FNR { A[$1]=1 ; next } $1 in A && $2 == "asn" { 
printf("AS%s\n", $3) }' cc.txt | sort | md>
    9ab3dbfbff5746f059cdb35221ff73b1

$ find . -type f -name "[12].txt" -print0 | xargs -0 -n 1 -P 1 -- 
~/go/bin/goawk -F '|' 'NR==FNR { A[$1]=1 ; next } $1 in A && $2 == "asn" { 
printf("AS%s\n", $3) }' cc.txt | sort | md
    9ab3dbfbff5746f059cdb35221ff73b1

I've racked my brain and the internet for hours, I've tested and toiled, and 
I'm left thoroughly perplexed. I now humbly ask the fine folks here in OpenBSD 
Land for guidance, insight or suggestions.

As always, is this a bug, or am I holding it wrong?

Regards,

Jordan



1.txt.gz
Description: 

Re: 6.9 Current amd64 xfce seems to freeze and not respond to mouse clicks or keystrokes

2021-04-10 Thread Jordan Geoghegan



On 4/10/21 2:48 PM, gwes wrote:
>
>
> On 4/10/21 5:22 PM, Tom Smyth wrote:
>> Hello,
>>
>> 1) issue does not occur with fvwm or with chrome running in fvwm
>>
>> so the issue seems to be confined to xfce, and I was running  just 1
>> xfce terminal session
>> 2) (so the issue is not related to chromium)
>>
>> Thanks
>>
>>
>> O
>> -- 
>> Kindest regards,
>> Tom Smyth.
>>
> Hi Tom,
> Some application that you can't see is grabbing focus and not letting go.
>
> On another OS using xfce (XUbuntu) the screensaver sometimes causes something
> extremely similar. Mouse cursor moves but nothing else responds.
> The workaround is to use control-alt-F1 to get a
> plain console and ps -ax | grep screen then doas kill .
>
> If it's not a screensaver it's almost always a second browser copy.
> I just find likely greedy candidates in the ps and kill until the problem 
> goes away.
>
> If the X server won't let you use control-alt-Fx to change screens you'll
> have to ssh in.
>
> Geoff Steckel
>

That xfce screen saver issue also seems to exists on OpenBSD 6.8, as I've 
triggered it a number of times on my desktop. I've found that even with the 
locked screen (mouse can move and nothing else as described) I can still  enter 
my password to unlock the screen saver, and then things usually work as 
expected -- if not, switching terminals with ctrl+alt+f1 and back with 
ctrl+alt+f5 seems to do the trick once the screen saver has been unlocked with 
my users password, with no need to kill the screen saver process.

Regards,

Jordan




Re: blacklistd analogue

2021-03-24 Thread Jordan Geoghegan


On 3/24/21 11:48 AM, Peter Nicolai Mathias Hansteen wrote:
>> 24. mar. 2021 kl. 19:33 skrev jeanpierre 
>> :
>>
>> Does there exist an OpenBSD analogue for FreeBSD's blacklistd daemon?
>>
>> For the sake of completeness: blacklistd is a daemon that, using pf
>> anchors, blocks connections from abusive hosts to parctiular services
>> (e.g. sshd) until they start behaving themselves again.
>>
>> I find it very useful for timming down log files.
> Not in the base system but you might want to take a peek at pf-badhosts 
> (described among other places in this OpenBSD Journal article 
> https://undeadly.org/cgi?action=article;sid=20210119113425 
> ) which should be 
> fairly easy to adapt to using more or other sources such as the bsdly.net 
>  feed maintained mainly by kind robots under supervision 
> by yours truly (see 
> https://bsdly.blogspot.com/2018/08/badness-enumerated-by-robots.html 
>  and 
> links therein and in the proximity)
>
> Cheers,
> Peter
>
> —
> Peter N. M. Hansteen, member of the first RFC 1149 implementation team
> http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/
> "Remember to set the evil bit on all malicious network traffic"
> delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.
>
>

I use Peter's "bruteforcers" list on my personal pf-badhost installs. 
pf-badhost will happily parse and ingest the IPv4 and IPv6 data in Peter's 
blocklists.

Just use the '-l' option to include an additional URL. Something like this 
should work to include his lists:

    $ pf-badhost -O openbsd -l 'https://www.bsdly.net/~peter/bruteforcers.txt'

I intend to include this among the default lists in the next release -- the 
only reason it wasn't included in v0.5 was because I discovered it too late.

Happy bot blocking!

Regards,

Jordan



Re: Attempting to use Brother DCP-L2510D printer on BSD

2021-02-24 Thread Jordan Geoghegan


On 2/24/21 9:43 AM, Sylvain S wrote:
> Hello,
>
> I see that, if I understand correctly, you have been
> able to make this printer work on ArchLinux from the
> officially provided RPM driver package.
> https://aur.archlinux.org/packages/brother-dcp-l2510d/
>
> May I ask, what kind of porting work did that require ?
> I have acquired this model from Brother and have not been
> successful at making it work/print yet.
> For reference, here is the thread on the French community forums
> https://forum.openbsd.fr.eu.org/showthread.php?tid=2846
>
> I also note that the printer is absent from the foomatic
> database maintained by Fedora. This seems strange as a
> driver package is available for their distribution format,
> but perhaps I am missing something here ?
> https://src.fedoraproject.org/repo/extras/foomatic-db/
>
> Finally, I notice from the documentation (to the mailing list's attention) :
>
> The Foomatic framework supports direct printing which does not rely on CUPS 
> (nor
> any spooler).
>
> Complete documentation can be found at:
>     
> https://wiki.linuxfoundation.org/openprinting/database/nospoolerdocumentation
>
> I would gladly do that, but am still clueless as of now
>
> Thanks to anyone who would help
>

It feels like we repeat this thread every couple months...

If your printer supports postscript (as yours and most other Brother brand 
laser printers seem to) you can use something like this in your /etc/printcap 
file to print postscript data directly to the printer:
...
lp|remote line printer:\
    
:lp=:rm=172.20.200.5:rp=lp:sd=/var/spool/output/lpd:lf=/var/log/lpd-errs:
...
Then check out stuff like the 'pdftops' utility etc for converting documents to 
postscript. With a bit of fiddling, LibreOffice can be configured to print raw 
postscript to the device as well. No drivers or other garbage needed. Everybody 
always wants to connect to their printer over USB, just don't -- its annoying. 
If you get your postscript printer setup on the network like this, then you can 
move forward into broad sunlit uplands.

Printing truly can be as easy as '$ lpr mydoc.ps'

Regards,

Jordan



Re: relayd + pfsync

2021-02-03 Thread Jordan Geoghegan



On 2/1/21 8:20 PM, Kapetanakis Giannis wrote:
> On 02/02/2021 05:18, Jordan Geoghegan wrote:
>> Hello,
>>
>> I had a question about using relayd with pfsync.
>>
>> I have a small gateway/load-balancer set up with relayd, carp and pfsync 
>> plus BGPd for IP failover, and everything is working great. I was pleasantly 
>> surprised at how easy it was to get pfsync tunnelled over wireguard. Things 
>> failover perfectly, and I'm happy as a clam.
>>
>> I however do have a question about some pfsync/relayd details that I'm not 
>> fully clear on:
>>
>> With all the plumbing being done with relayd and all the associated 
>> TCP/TLS/HTTP(s) checks it's doing, it ends up setting up and tearing down a 
>> decent number of connections on a recurring basis. I know in PF you can use 
>> the "no-sync" keyword to prevent states created by certain rules from being 
>> synced across the wire, but I haven't found a way to do this with 
>> rules/states generated by relayd.
>>
>> It's probably largely irrelevant in the grand scheme of things, but I found 
>> it slightly irritating having hundreds or thousands of state table entries 
>> experiencing constant churn while being synced over the wire. Having the 
>> noise from the relayd connectivity checks syncing back and forth makes using 
>> tcpdump on a pfsync interface much less convenient. All these state table 
>> entries will never be used should the machine fail-over, as all the 
>> connectivity checks are initiated from the local IP address, rather that the 
>> CARP address.
>>
>> So I guess what I'm trying to ask is: Is there a way to have relayd not sync 
>> it's TCP/TLS/etc connectivity checks via pfsync?
>>
>> I was hoping to get a sanity check here so I can confirm weather or not I'm 
>> totally off base here.
>>
>> I currently have "keep state (no-sync)" peppered throughout my config for 
>> rules I want excluded from pfsync, as the pf config is quite simple. Maybe 
>> I'm missing something obvious, but is there a "sync" option? ie the ability 
>> to manually specify exactly which rules/states you want synced?
>>
>> Would some sort of rule like "pass out on $int_if proto tcp to any user 
>> _relayd keep state (no-sync)" do what I want, or would that also catch the 
>> traffic I'm trying to load balance as well?
>>
>> Any insight or advice would be much appreciated.
>>
>> P.S  Sorry for the wall of text
>>
>> Regards,
>>
>> Jordan
>
> Hi,
>
> As you said, you can use the no-sync.
>
> Relayd checks don't create any pf rules. Only the listen creates rules
> pfctl -sr -a'relayd/ldap'
>
> pass in quick on rdomain 0 inet proto tcp from any to x.x.x.x port = 636 
> flags S/SA keep state (tcp.established 4200) tag RELAYD_ldap rdr-to  
> port 1636 least-states sticky-address
>
> local checks from LB to hosts can have the no-sync.
> I have these in my config
>
> # checks from LB
> pass out quick on $ldap_if proto tcp from ($ldap_if) to ($ldap_if:network) 
> port {1389, 1636} keep state (no-sync)
>
> Maybe you have another rule (out on $ldap_if) before, that allows the traffic?
>
> I handle incoming traffic (to LB) with pftag on relayd.conf and I 
> specifically allow them on out direction.
>
> # client rules
> pass out quick on $ldap_if tagged RELAYD_ldap keep state (tcp.established 
> 4200)
>
> G
>
>

Hello,

Thanks for the sanity check, I've got everything working as expected now!

I need to learn to step away from the keyboard when I'm getting tired and 
frustrated. The answer seems obvious in retrospect, but at least its figured 
out now.

Regards,

Jordan




relayd + pfsync

2021-02-01 Thread Jordan Geoghegan
Hello,

I had a question about using relayd with pfsync.

I have a small gateway/load-balancer set up with relayd, carp and pfsync plus 
BGPd for IP failover, and everything is working great. I was pleasantly 
surprised at how easy it was to get pfsync tunnelled over wireguard. Things 
failover perfectly, and I'm happy as a clam.

I however do have a question about some pfsync/relayd details that I'm not 
fully clear on:

With all the plumbing being done with relayd and all the associated 
TCP/TLS/HTTP(s) checks it's doing, it ends up setting up and tearing down a 
decent number of connections on a recurring basis. I know in PF you can use the 
"no-sync" keyword to prevent states created by certain rules from being synced 
across the wire, but I haven't found a way to do this with rules/states 
generated by relayd.

It's probably largely irrelevant in the grand scheme of things, but I found it 
slightly irritating having hundreds or thousands of state table entries 
experiencing constant churn while being synced over the wire. Having the noise 
from the relayd connectivity checks syncing back and forth makes using tcpdump 
on a pfsync interface much less convenient. All these state table entries will 
never be used should the machine fail-over, as all the connectivity checks are 
initiated from the local IP address, rather that the CARP address.

So I guess what I'm trying to ask is: Is there a way to have relayd not sync 
it's TCP/TLS/etc connectivity checks via pfsync?

I was hoping to get a sanity check here so I can confirm weather or not I'm 
totally off base here.

I currently have "keep state (no-sync)" peppered throughout my config for rules 
I want excluded from pfsync, as the pf config is quite simple. Maybe I'm 
missing something obvious, but is there a "sync" option? ie the ability to 
manually specify exactly which rules/states you want synced?

Would some sort of rule like "pass out on $int_if proto tcp to any user _relayd 
keep state (no-sync)" do what I want, or would that also catch the traffic I'm 
trying to load balance as well?

Any insight or advice would be much appreciated.

P.S  Sorry for the wall of text

Regards,

Jordan






Re: File this bug, or not?

2021-01-18 Thread Jordan Geoghegan



On 1/18/21 2:47 PM, Eric Zylstra wrote:
> Misc,
>
> I’ve set up a 6 drive RAID-5.  Just for the experience of degrading and 
> rebuilding the RAID, I popped a drive out.  Within a few seconds the machine 
> kerneled and dropped into ddb.  Is there any chance this would be expected 
> considering the machine’s SATA is not hot-swappable?
>
> I’m looking into setting up a serial connection so I can capture the debut 
> output (I already have photos of the traces for all 8 CPU, but would like to 
> give serial output instead).  I would not file a report if this behavior 
> falls into “not great, but expected”.
>
> Thanks,
>
> EZ
>

Just thought I'd chip in here too FWIW:

I've never successfully hot swapped a drive with OpenBSD before. I have 
hardware that does it fine on Linux, but fails on OpenBSD. I haven't caused the 
kernel to panic when pulling a drive, but the OS fails to detect any newly 
attached SATA or SAS drives. It's certainly caused some frustration when trying 
to rebuild a RAID array on a production machine. Maybe I just have wonky 
hardware, but I've tried this on a number of releases, on several different 
pieces of hardware, on several different arches. I have no solution to offer, 
just thought I'd share my experience with hot swapping drives on OpenBSD.

Regards,

Jordan



pf-badhost and unbound-adblock v0.5 released

2021-01-10 Thread Jordan Geoghegan
Hey folks,

Since I've been getting a lot of emails about this, I just thought I'd drop a 
line here and let you know that I've released pf-badhost and unbound-adblock 
version 0.5.

You can see the release pages here:

pf-badhost is a fast, in-kernel, bi-directional network filtering utility 
powered by the PF firewall.
https://geoghegan.ca/pfbadhost.html

unbound-adblock is a fast, flexible, easy to use DNS firewall utility.
https://geoghegan.ca/unbound-adblock.html

This will be the last time I advertise my wares on here as I have established a 
notification service -- just send an email to "annou...@geoghegan.ca" to 
subscribe.

One new feature that I would like to highlight is that unbound-adblock now 
supports florian@'s excellent unwind(8) resolver as an optional backend. I've 
been running this on my laptop for about a month now and it's working quite 
nicely.

Thank you to everyone in the the OpenBSD community who have on many occasions, 
kindly assisted with development in one way or the other -- I am truly grateful.

Regards,

Jordan Geoghegan



Re: cmp(1) '-s' flag ignoring byte offset argument?

2021-01-10 Thread Jordan Geoghegan



On 1/9/21 1:59 AM, Otto Moerbeek wrote:
> On Sat, Jan 09, 2021 at 12:05:31AM -0800, William Ahern wrote:
>
>> On Fri, Jan 08, 2021 at 07:09:01PM -0800, Jordan Geoghegan wrote:
>>> Hey folks,
>>>
>>> I've noticed some surprising behaviour from cmp(1) when using the '-s'
>>> flag.
>>>
>>> It appears that cmp -s is ignoring the byte offset arguments I'm giving
>>> it.
>> 
>>> Not sure what to make of this, I noticed this same behaviour on
>>> DragonflyBSD and FreeBSD, so maybe I'm just missing something obvious.
>>> This certainly caused some frustration before I figured out what was going
>>> on.
>> The bug seems to be in the short-circuit optimization for regular files[1]:
>>
>> void
>>   c_regular(int fd1, char *file1, off_t skip1, off_t len1,
>>   int fd2, char *file2, off_t skip2, off_t len2)
>>   {
>>  u_char ch, *p1, *p2;
>>  off_t byte, length, line;
>>  int dfound;
>>   
>>  if (sflag && len1 != len2)
>>  exit(1);
>>   
>>  if (skip1 > len1)
>>  eofmsg(file1);
>>  len1 -= skip1;
>>  if (skip2 > len2)
>>  eofmsg(file2);
>>  len2 -= skip2;
>>
>> The short-circuit should probably be moved below the subsequent chunk of
>> code (i.e. below `len2 -= skip2`). The eofmsg function already obeys sflag,
>> so it'll be quiet.[2] Doing this works for me. See patch at end of message.
>>
>> Interestingly, DragonflyBSD and FreeBSD already do it this way[3][4], yet I
>> can confirm FreeBSD still has the problem. (DragonflyBSD has nearly
>> identical code.) But that implementation duplicates the short-circuit, along
>> with the bug of not accounting for skip1 and skip2, in cmp.c as part of
>> implementing the -z flag[5]:
>>
>>  if (special)
>>  c_special(fd1, file1, skip1, fd2, file2, skip2);
>>  else {
>>  if (zflag && sb1.st_size != sb2.st_size) {
>>  if (!sflag)
>>  (void) printf("%s %s differ: size\n",
>>  file1, file2);
>>  exit(DIFF_EXIT);
>>  }
>>  c_regular(fd1, file1, skip1, sb1.st_size,
>>  fd2, file2, skip2, sb2.st_size);
>>  }
>>  exit(0);
>>
>> It appears that the June 20, 2000 fix to the short-circuit in regular.c
>> wasn't recognized during the July 14, 2000 -z feature addition.[6][7]
>>
>> [1] https://cvsweb.openbsd.org/src/usr.bin/cmp/regular.c?rev=1.12
>> [2] https://cvsweb.openbsd.org/src/usr.bin/cmp/misc.c?rev=1.7
>> [3] 
>> https://gitweb.dragonflybsd.org/dragonfly.git/blob/4d4f84f:/usr.bin/cmp/regular.c
>> [4] 
>> https://svnweb.freebsd.org/base/head/usr.bin/cmp/regular.c?revision=344551
>> [5] 
>> https://svnweb.freebsd.org/base/head/usr.bin/cmp/cmp.c?revision=344551=markup#l193
>> [6] 
>> https://svnweb.freebsd.org/base/head/usr.bin/cmp/regular.c?revision=61883=markup
>> [7] 
>> https://svnweb.freebsd.org/base/head/usr.bin/cmp/cmp.c?view=markup=63157
>>
>> --- regular.c6 Feb 2015 23:21:59 -   1.12
>> +++ regular.c9 Jan 2021 07:51:13 -
>> @@ -51,15 +51,15 @@ c_regular(int fd1, char *file1, off_t sk
>>  off_t byte, length, line;
>>  int dfound;
>>  
>> -if (sflag && len1 != len2)
>> -exit(1);
>> -
>>  if (skip1 > len1)
>>  eofmsg(file1);
>>  len1 -= skip1;
>>  if (skip2 > len2)
>>  eofmsg(file2);
>>  len2 -= skip2;
>> +
>> +if (sflag && len1 != len2)
>> +exit(1);
>>  
>>  length = MINIMUM(len1, len2);
>>  if (length > SIZE_MAX) {
>>
> I came to the same diff independently. In the meantime it has been committed.
>
>   -Otto
>

Hi Otto and William,

Thanks for confirming that this is a bug, and also for the fixes!

Regards,

Jordan



cmp(1) '-s' flag ignoring byte offset argument?

2021-01-08 Thread Jordan Geoghegan
Hey folks,

I've noticed some surprising behaviour from cmp(1) when using the '-s' flag.

It appears that cmp -s is ignoring the byte offset arguments I'm giving it.

I don't want to waste time babbling, so here's an example snippet to show what 
I'm talking about:

#!/bin/sh

echo 'my line' > /tmp/1.txt
echo 'my other line' >> /tmp/1.txt
echo 'same same' >> /tmp/1.txt

echo 'my differnt line' > /tmp/2.txt
echo 'my other different line' >> /tmp/2.txt
echo 'same same' >> /tmp/2.txt

# Determine byte offsets (we only want to compare lines >= 3)
offset1="$(head -2 /tmp/1.txt | wc -c)"
offset2="$(head -2 /tmp/2.txt | wc -c)"

# Compare files and show exit code
cmp /tmp/1.txt /tmp/2.txt "$offset1" "$offset2"
printf '\nReturn code = %s\n' "$?"

cmp -s /tmp/1.txt /tmp/2.txt "$offset1" "$offset2"
printf '\nReturn code with "-s" = %s\n' "$?"

As you can see, 'cmp -s' returns an exit code of '1', unlike cmp without the 
'-s' which returns '0'.

Not sure what to make of this, I noticed this same behaviour on DragonflyBSD 
and FreeBSD, so maybe I'm just missing something obvious. This certainly caused 
some frustration before I figured out what was going on.

Regards,

Jordan



An ode to OpenBSD

2020-12-22 Thread Jordan Geoghegan
This isn't actually an ode, as I can't rhyme things good, but I just 
wanted to thank the OpenBSD folks for their fantastic engineering.


I had a major power outage and network disruption at my hosting provider 
yesterday, and all but one of my relayd load balancers was knocked out, 
as well as much of my httpd cluster.


Even while Zabbix was blowing up my phone with alerts, thanks to the 
magic of CARP+pfsync+relayd, all my services remained available, and a 
disaster was largely turned into a non-event.


Thank you for creating the greatest software project of all time.

Regards,

Jordan



Re: OpenBSD Monitor Sleep No Response

2020-12-20 Thread Jordan Geoghegan




On 12/20/20 6:26 PM, ben wrote:

Hello, misc;

I've been having an issue with my OpenBSD install, specifically when the system
turns off the monitor after a period of no use. After the monitor goes blank I
can't use the start using the machine and must restart, that is after keypress
and mouse movement the system does not show anything on the monitor. I suspect
something is wrong due to hardware. I've checked the logs, nothing seems to be
off. I've turned off apmd as to not interfer with power management and still no
response after the monitor goes to sleep.

Here's a list of the hardware:

  - AMD Ryzen 5 3400G Processor
  - Asus Prime B550M-A/CSM Motherboard
  - Radeon RX 580 POLARIS10 GPU

Has anyone else experience any issues with like this? Is there still no support
for polaris GPUs? Thank you in advance.


Ben Raskin



I've had this issue as well with a Ryzen machine with an R9290x graphics 
card (ie not supported by AMDGPU). My monitor is using DisplayPort but 
the same behaviour is present when using HDMI monitors as well.


I've found the only way I can get output to the screen again is by 
switching consoles with CTRL+ALT+F1 and then switching back to the xorg 
console with CTRL+ALT+F5.


Not sure what the cause is, but I've always assumed it was an issue the 
the radeon driver.


Regards,

Jordan



Re: Potential dig bug?

2020-12-17 Thread Jordan Geoghegan




On 12/16/20 11:19 PM, Otto Moerbeek wrote:

On Wed, Dec 16, 2020 at 02:37:19PM -0800, Jordan Geoghegan wrote:


Hi folks,

I've found some surprising behaviour in the 'dig' utility. I've noticed that
dig doesn't seem to support link local IPv6 addresses. I've got unbound
listening on a link local IPv6 address on my router and all queries seem to
be working. I'm advertising this DNS info with rad, and I confirmed with
tcpdump that my devices such as iPhones, Macs, Windows, Linux desktops etc
are all properly querying my unbound server over IPv6.

dhclient doesn't seem to allow you to specify an IPv6 address in it's
'supersede'  options, so I manually edited my OpenBSD desktops resolv.conf
to specify the IPv6 unbound server first. Again, I confirmed with tcpdump
that my desktop was properly querying the unbound server over IPv6 (ie
Firefox, ping, ssh etc all resolved domains using this server).

I used 'dig' to make a query, and I noticed it was ignoring my link local
IPv6 nameserver in my resolv.conf. I'll save you guys the long form Ted talk
here and just make my point:

$ cat resolv.conf
    nameserver fe80::f29f:c2ff:fe17:b8b2%em0
    nameserver 2606:4700:4700::
    lookup file bind
    family inet6 inet4

$ dig google.ca
    [snip]
    ;; Query time: 12 msec
    ;; SERVER: 2606:4700:4700::#53(2606:4700:4700::)
    [snip]

There's a bit of a delay as it waits for a time out, and then it falls back
to the cloudflare IPv6 server.

I tried specifying the server with '@' as well as specifying source
IP/interface with '-I' to no avail. It seems dig really doesn't like the
'fe80::%em0' notation, as  '@' and '-I' worked fine when used without a
link-local address.

Is this a bug or a feature? Am I just doing something stupid? Any insight
would be appreciated.

I think it is a bug and I can reproduce. Will invesigate deeper later.

-Otto



Hi Otto,

Thanks for looking into this! I took Bodie's advice and tested nslookup 
and host, and they both seem to have the same behaviour as dig.


Regards,

Jordan



Re: Potential dig bug?

2020-12-16 Thread Jordan Geoghegan




On 12/16/20 2:37 PM, Jordan Geoghegan wrote:

Hi folks,

I've found some surprising behaviour in the 'dig' utility. I've 
noticed that dig doesn't seem to support link local IPv6 addresses. 
I've got unbound listening on a link local IPv6 address on my router 
and all queries seem to be working. I'm advertising this DNS info with 
rad, and I confirmed with tcpdump that my devices such as iPhones, 
Macs, Windows, Linux desktops etc are all properly querying my unbound 
server over IPv6.


dhclient doesn't seem to allow you to specify an IPv6 address in it's 
'supersede'  options, so I manually edited my OpenBSD desktops 
resolv.conf to specify the IPv6 unbound server first. Again, I 
confirmed with tcpdump that my desktop was properly querying the 
unbound server over IPv6 (ie Firefox, ping, ssh etc all resolved 
domains using this server).


I used 'dig' to make a query, and I noticed it was ignoring my link 
local IPv6 nameserver in my resolv.conf. I'll save you guys the long 
form Ted talk here and just make my point:


$ cat resolv.conf
   nameserver fe80::f29f:c2ff:fe17:b8b2%em0
   nameserver 2606:4700:4700::
   lookup file bind
   family inet6 inet4

$ dig google.ca
   [snip]
   ;; Query time: 12 msec
   ;; SERVER: 2606:4700:4700::#53(2606:4700:4700::)
   [snip]

There's a bit of a delay as it waits for a time out, and then it falls 
back to the cloudflare IPv6 server.


I tried specifying the server with '@' as well as specifying source 
IP/interface with '-I' to no avail. It seems dig really doesn't like 
the 'fe80::%em0' notation, as  '@' and '-I' worked fine when used 
without a link-local address.


Is this a bug or a feature? Am I just doing something stupid? Any 
insight would be appreciated.


Regards,

Jordan


Sorry for the double mail, I hit send too early...

Woops, I failed to make the key point here:

I checked with tcpdump and confirmed that dig does not even attempt to 
query the IPv6 link local DNS server, even though it reports a timeout - 
ie dig sends no traffic over the wire destined to that address:


; <<>> dig 9.10.8-P1 <<>> @fe80::f29f:c2ff:fe17:b8b2%em0 google.ca
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

Regards,
Jordan



Potential dig bug?

2020-12-16 Thread Jordan Geoghegan

Hi folks,

I've found some surprising behaviour in the 'dig' utility. I've noticed 
that dig doesn't seem to support link local IPv6 addresses. I've got 
unbound listening on a link local IPv6 address on my router and all 
queries seem to be working. I'm advertising this DNS info with rad, and 
I confirmed with tcpdump that my devices such as iPhones, Macs, Windows, 
Linux desktops etc are all properly querying my unbound server over IPv6.


dhclient doesn't seem to allow you to specify an IPv6 address in it's 
'supersede'  options, so I manually edited my OpenBSD desktops 
resolv.conf to specify the IPv6 unbound server first. Again, I confirmed 
with tcpdump that my desktop was properly querying the unbound server 
over IPv6 (ie Firefox, ping, ssh etc all resolved domains using this 
server).


I used 'dig' to make a query, and I noticed it was ignoring my link 
local IPv6 nameserver in my resolv.conf. I'll save you guys the long 
form Ted talk here and just make my point:


$ cat resolv.conf
   nameserver fe80::f29f:c2ff:fe17:b8b2%em0
   nameserver 2606:4700:4700::
   lookup file bind
   family inet6 inet4

$ dig google.ca
   [snip]
   ;; Query time: 12 msec
   ;; SERVER: 2606:4700:4700::#53(2606:4700:4700::)
   [snip]

There's a bit of a delay as it waits for a time out, and then it falls 
back to the cloudflare IPv6 server.


I tried specifying the server with '@' as well as specifying source 
IP/interface with '-I' to no avail. It seems dig really doesn't like the 
'fe80::%em0' notation, as  '@' and '-I' worked fine when used without a 
link-local address.


Is this a bug or a feature? Am I just doing something stupid? Any 
insight would be appreciated.


Regards,

Jordan



Re: Another potential ksh bug?

2020-12-11 Thread Jordan Geoghegan

Hi Vadim,

Thanks for looking into this, all I ever wanted was an ancient pdksh bug 
fix for Christmas.


Regards,

Jordan



On 12/7/20 10:47 AM, Vadim Zhukov wrote:

For me, this is a definite bug. I've opted my students to fix this
bug, so unless there's a hurry, there must be a fix till the end of
December. :)

пн, 7 дек. 2020 г. в 07:43, Jordan Geoghegan :

Hello again,

I was playing around with ksh array syntax and its behaviour when set as
read-only. In my testing I noticed that ksh will allow you to overwrite
the first element of a read-only array. Example snippet:

#!/bin/ksh
arr[0]=val1
arr[1]=val2
readonly arr
echo "${arr[@]}"
arr=yikes
echo "${arr[@]}"

I tested a few other shells, and this bug does exists in the original
pdksh and is also present in zsh. This bug is not present in ksh93, mksh
or bash, where they abort when trying to modify the read-only array.

I don't have access to a proper ksh88 shell, but it would be nice if
someone could confirm its behaviour.

I was just hoping someone could confirm if this is intended behaviour,
or if it's a bug.

Regards,

Jordan







Re: Another potential ksh bug?

2020-12-11 Thread Jordan Geoghegan

On 12/7/20 10:14 PM, Noth wrote:


On 07/12/2020 05:41, Jordan Geoghegan wrote:

Hello again,

I was playing around with ksh array syntax and its behaviour when set 
as read-only. In my testing I noticed that ksh will allow you to 
overwrite the first element of a read-only array. Example snippet:


#!/bin/ksh
arr[0]=val1
arr[1]=val2
readonly arr
echo "${arr[@]}"
arr=yikes
echo "${arr[@]}"

I tested a few other shells, and this bug does exists in the original 
pdksh and is also present in zsh. This bug is not present in ksh93, 
mksh or bash, where they abort when trying to modify the read-only 
array.


I don't have access to a proper ksh88 shell, but it would be nice if 
someone could confirm its behaviour.


I was just hoping someone could confirm if this is intended 
behaviour, or if it's a bug.


Regards,

Jordan


Hi,

  I tries this with AMIX (UNIX SVR4 for Commodore Amiga) v2.1 which 
presumably is using ksh88 as it hasn't been updated since circa 1992 
and it does abort at the same place. Screenshot here: 
http://casper.nineinchnetworks.ch/images/kshtest.png .


Cheers,

Noth


Hi Noth,

Sorry for the late reply, I've been AFK for several days.

Thanks for testing that out and confirming ksh88's behaviour!

Regards,

Jordan



Another potential ksh bug?

2020-12-06 Thread Jordan Geoghegan

Hello again,

I was playing around with ksh array syntax and its behaviour when set as 
read-only. In my testing I noticed that ksh will allow you to overwrite 
the first element of a read-only array. Example snippet:


#!/bin/ksh
arr[0]=val1
arr[1]=val2
readonly arr
echo "${arr[@]}"
arr=yikes
echo "${arr[@]}"

I tested a few other shells, and this bug does exists in the original 
pdksh and is also present in zsh. This bug is not present in ksh93, mksh 
or bash, where they abort when trying to modify the read-only array.


I don't have access to a proper ksh88 shell, but it would be nice if 
someone could confirm its behaviour.


I was just hoping someone could confirm if this is intended behaviour, 
or if it's a bug.


Regards,

Jordan



Re: Potential ksh bug?

2020-11-16 Thread Jordan Geoghegan




On 11/16/20 10:36 PM, Bodie wrote:



On 17.11.2020 05:04, Jordan Geoghegan wrote:

Hello,

I'm not sure if this is a bug, or if it's just a pdksh thing, but I
stumbled upon some interesting behaviour when I was tinkering around
with quoting and using a poor mans array:

test=$(cat <<'__EOT'
# I'll choose not to close this quote
other_stuff
__EOT
)

echo "$test"


When I run this command on ash, dash, yash, bash, zsh or ksh93 I get
the following output:

# I'll choose not to close this quote
other_stuff

But when I run it on ksh from base or any pdksh derivative it throws
an error about an unclosed quote:

test.sh[8]: no closing quote

This snippet works on every POSIX-y shell in the ports tree, and fails
on every pdksh variant I tried, including on NetBSD and DragonflyBSD
as well.  I don't have the requisite esoteric knowledge regarding
pdksh's internal quoting logic, so I'm hoping one of the gurus here
can determine whether this is a bug or if I'm just doing something
annoying.

Any insight that can be provided would be much appreciated.



What exactly are you trying to achieve?

If you will look in sh(1) for 'Command expansion' then there are defined
rules and your form is not between them.

So error message about missing closing quote is actually proper behavior.

Plus I will not be so sure that bash is behaving properly:

$ bash -x test.sh
++ cat
+ test='
# I'\''ll choose not to close this quote
other_stuff'
+ echo '#' 'I'\''ll' choose not to close this quote other_stuff
# I'll choose not to close this quote other_stuff
$

Notice how one single quote is here "test='" abd second one is in
"other_stuff'"

You can compare results between various shells with ktrace/ktruss too

$ bash --version | head -1
GNU bash, version 5.0.18(1)-release (x86_64-unknown-openbsd6.8)
$

If you want to read the file then you do not need interaction and if
you want to get input from user then use 'read'

As well it is good idea to avoid reserved words as a names for 
variables ;-)

(test)


Regards,

Jordan




I'm not trying to achieve anything in particular, I was just testing out 
differences in behaviour between different shells. I just noticed that 
pdksh variants handled this differently than any other shell I tested in 
the ports tree.


Regards,

Jordan



Potential ksh bug?

2020-11-16 Thread Jordan Geoghegan

Hello,

I'm not sure if this is a bug, or if it's just a pdksh thing, but I 
stumbled upon some interesting behaviour when I was tinkering around 
with quoting and using a poor mans array:


test=$(cat <<'__EOT'
# I'll choose not to close this quote
other_stuff
__EOT
)

echo "$test"


When I run this command on ash, dash, yash, bash, zsh or ksh93 I get the 
following output:


# I'll choose not to close this quote
other_stuff

But when I run it on ksh from base or any pdksh derivative it throws an 
error about an unclosed quote:


test.sh[8]: no closing quote

This snippet works on every POSIX-y shell in the ports tree, and fails 
on every pdksh variant I tried, including on NetBSD and DragonflyBSD as 
well.  I don't have the requisite esoteric knowledge regarding pdksh's 
internal quoting logic, so I'm hoping one of the gurus here can 
determine whether this is a bug or if I'm just doing something annoying.


Any insight that can be provided would be much appreciated.

Regards,

Jordan



Re: APU4 hardware network interfaces tied together

2020-11-15 Thread Jordan Geoghegan




On 11/15/20 12:25 PM, Mihai Popescu wrote:

Hello,

In the scenario of building a router with APU4, one interface is for wan,
the rest of three are free to use.
What is the most sane and performance wise ( CPU load, interface load,
etc.) way to tie together the remaining three interfaces as a switch, and
avoid using one IP class per interface?
Is it better to use one for lan, leave the remaining two unused and cascade
a dumb switch for other lan connections?

Thank you.


I wouldn't recommend putting the remaining ports into a bridge 
configuration as that will force the interfaces into promiscuous mode, 
and cause higher CPU load. It would be better to just run the LAN off of 
a switch connected to a single port on the APU as that will allow LAN 
traffic to flow without the APU having to touch every single packet. If 
you wanted to be pedantic, an argument could also be made that using a 
single interface would also lend itself to maximally effective interrupt 
coalescing.


Regards,

Jordan



Re: OBSD 6.8 vlan communication issues

2020-11-12 Thread Jordan Geoghegan




On 11/11/20 3:06 PM, len zaifman wrote:
I am setting up a new system as a firewall using OpenBSD 6.8 current 
-uname -a

OpenBSD fw1.lfz.net 6.8 GENERIC.MP#175 amd64.

I have 3 vlans 70,77,79 on  the firewall using two em devices, em0 and 
em1, in an aggregation to serve these vlans.



There is a Unifi switch which has 2 ports (where em0,em1 are attached) 
set up to pass tagged vlans 70,77,79. The switch ip is 10.10.70.3.


I have a linux host setup on vans 70,77,79 and at address 77 - 
10.10.70.77, 10.10.77.77,10.10.79.77.



So far i cannot communicate over the vlans. Before I vlanned these 
subnets : ie only vlan 1 everywhere - communication worked fine.


So i do not believe there is a physical issue. The issues arose with 
the introduction of the vlans. Is there a configuration issue that 
anyone can spot?



Thank you for any help you can give.

Evidence:

ping on the firewall works locally

for n in 0 7 9 ; do ping -c 2 10.10.7${n}.1 ; done
PING 10.10.70.1 (10.10.70.1): 56 data bytes
64 bytes from 10.10.70.1: icmp_seq=0 ttl=255 time=0.037 ms
64 bytes from 10.10.70.1: icmp_seq=1 ttl=255 time=0.025 ms

--- 10.10.70.1 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.025/0.031/0.037/0.006 ms
PING 10.10.77.1 (10.10.77.1): 56 data bytes
64 bytes from 10.10.77.1: icmp_seq=0 ttl=255 time=0.038 ms
64 bytes from 10.10.77.1: icmp_seq=1 ttl=255 time=0.025 ms

--- 10.10.77.1 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.025/0.031/0.038/0.006 ms
PING 10.10.79.1 (10.10.79.1): 56 data bytes
64 bytes from 10.10.79.1: icmp_seq=0 ttl=255 time=0.038 ms
64 bytes from 10.10.79.1: icmp_seq=1 ttl=255 time=0.025 ms

--- 10.10.79.1 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.025/0.032/0.038/0.007 ms


ping to the switch does not work

ping -c 2 10.10.70.3
PING 10.10.70.3 (10.10.70.3): 56 data bytes

--- 10.10.70.3 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss

ping to the linux host does not work.

ping -c 2 10.10.70.3
PING 10.10.70.3 (10.10.70.3): 56 data bytes

--- 10.10.70.3 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
[13:47:04] leonardz@fw1 etc>>for n in 0 7 9 ; do ping -c 2 
10.10.7${n}.77 ; done

PING 10.10.70.77 (10.10.70.77): 56 data bytes

--- 10.10.70.77 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
PING 10.10.77.77 (10.10.77.77): 56 data bytes

--- 10.10.77.77 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
PING 10.10.79.77 (10.10.79.77): 56 data bytes

--- 10.10.79.77 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss

I did the tests both with pfctl -e (enabled) and pfctl -d (disabled). 
It made no difference



The setup is described below

Here is the setup:

= hostname.aggr0
debug
trunkport em0
trunkport em1
up
inet 10.10.70.1/24
alias  10.10.77.1/24
alias  10.10.79.1/24


= hostname.em0
up

= hostname.em1
up


= hostname.vlan70
parent aggr0 vnetid 70
10.10.70.0/24

= hostname.vlan77
parent aggr0 vnetid 77
10.10.77.0/24

= hostname.vlan79
parent aggr0 vnetid 79
10.10.79.0/24


Ifconfig -A shows the vlans are setup

= aggr0
aggr0: flags=8847 mtu 1500
    lladdr fe:e1:ba:d0:f4:8c
    index 6 priority 0 llprio 7
    trunk: trunkproto lacp
    trunk id: [(8000,fe:e1:ba:d0:f4:8c,0006,,),
         (8000,e0:63:da:8e:78:d7,03E8,,)]
        em0 lacp actor system pri 0x8000 mac fe:e1:ba:d0:f4:8c, key 
0x6, port pri 0x8000 number 0x1
        em0 lacp actor state 
activity,aggregation,sync,collecting,distributing
        em0 lacp partner system pri 0x8000 mac e0:63:da:8e:78:d7, key 
0x3e8, port pri 0x1 number 0x9
        em0 lacp partner state 
activity,aggregation,sync,collecting,distributing

        em0 port active,collecting,distributing
        em1 lacp actor system pri 0x8000 mac fe:e1:ba:d0:f4:8c, key 
0x6, port pri 0x8000 number 0x2
        em1 lacp actor state 
activity,aggregation,sync,collecting,distributing
        em1 lacp partner system pri 0x8000 mac e0:63:da:8e:78:d7, key 
0x3e8, port pri 0x1 number 0xa
        em1 lacp partner state 
activity,aggregation,sync,collecting,distributing

        em1 port active,collecting,distributing
    groups: aggr
    media: Ethernet autoselect
    status: active
    inet 10.10.70.1 netmask 0xff00 broadcast 10.10.70.255
    inet 10.10.77.1 netmask 0xff00 broadcast 10.10.77.255
    inet 10.10.79.1 netmask 0xff00 broadcast 10.10.79.255

= em0
em0: flags=8843 mtu 1500
    lladdr fe:e1:ba:d0:f4:8c
    index 1 priority 0 llprio 3
    trunk: trunkdev aggr0
    media: Ethernet autoselect (1000baseT full-duplex)
    status: active

= em1
em1: flags=8843 mtu 1500
    lladdr fe:e1:ba:d0:f4:8c
    index 

Re: home printer

2020-09-17 Thread Jordan Geoghegan




On 2020-09-17 12:15, Greg Thomas wrote:

I've always been happy with the cheap Brother laser printers with ethernet,
even with just their version of Postscript.  But I believe they still sell
Postscript printers, too.

On Thu, Sep 17, 2020 at 10:07 AM Ingo Schwarze  wrote:


Hi Carson,

Carson Chittom wrote on Thu, Sep 17, 2020 at 09:51:45AM -0500:

Jan Stary  writes:

Can people please recommend a home laser printer
that is known to work well with OpenBSD?

I would like to avoid cups, and possibly a2ps
and foo* and if= and all that dance
- a printer that speaks postscript and is as easy as
lp:lp=/dev/lp:sd=/var/spool/output/lpd:lf=/var/log/lpd-errs:

HP at least used to (and I assume still do) make several decent
printers that spoke Postscript.

That answer used to be spot on until about the year 2000.  After
that, quality of HP laser printers went down the drain very rapidly.
One office i worked in decided in 2003 that the then more then five
year old HP LaserJet might die from old age soon and bought a new
one to be safe and not experience service disruption.  The old one
was left running, too, because why not, and printing traffic was
shared about evenly between the two because people tended to use
the one closest to their desk.

When the *successor* of the new one died from old age about six to
eight years later (i.e. when two of the new ones had worn out one
after the other, don't remember how long they lasted exactly, but
not longer than three or four years i think), the old one was still
going strong.  If i remember correctly, when the pre-2000 one finally
did die from old age, it was probably fifteen years old, if not
more, with continuous office use.

I doubt HP printers have become better again, but i'm not sure.


In particular, I've used the
CP1525nw in the past with OpenBSD.  Haven't tried it in a couple
years, though; none of my OpenBSD machines need to print, these
days.

Same here.  Currently, a Kyocera P2135dn is sitting on the desk here,
but i can't say whether it is good because i'm printing so little.

To the OP, what matters is a decent PostScript Processor
and a RJ45 Ethernet connector, then it will work with OpenBSD
no matter what.

Yours,
   Ingo




Greg has the right idea. I've had a great experience with my Brother 
MFC-L5700 laser printer. I can print Postscript and PDF documents to it 
directly using lpd/lpr in the OpenBSD base system, and the printer 
supports directly uploading scans to an FTP/SFTP server. It's quite nice 
being able to print and scan without messing around with drivers/cups 
and the dark arts magic required to make it work reliably.


I set my printer up by enabling lpd and adding a single line in my 
/etc/printcap file:


lp|remote line printer:\
:lp=:rm=192.0.2.5:rp=lp:sd=/var/spool/output/lpd:lf=/var/log/lpd-errs:



So far this setup has been totally bulletproof and has yet to fail me 
after over 5000 scanned pages and a couple thousand print jobs.


Regards,

Jordan





Re: Very slow clock in Debian vmm guest

2020-08-29 Thread Jordan Geoghegan
If you check the mailing list archives, you will see that this issue has 
been discussed extensively.


Dave Voutila has written a linux vmm kernel driver to work around some 
of the issues:


https://github.com/voutilad/virtio_vmmci

Regards,

Jordan

On 2020-08-28 20:48, Aaron Miller wrote:

I have a debian testing guest running in vmm(4) on my -current
system, and the internal clock is very slow. For example running
`sleep 3` takes about 10 seconds of real time to run. This is too
much for ntpd to correct, unfortunately.

Anyone know what the problem is and how I might go about fixing
it? Thanks!

--Aaron

OpenBSD 6.7-current (GENERIC.MP) #36: Sat Aug 22 11:27:03 MDT 2020
 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENE
RIC.MP
real mem = 16827916288 (16048MB)
avail mem = 16302870528 (15547MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xccbfd000 (65 entries)
bios0: vendor LENOVO version "N14ET37W (1.15 )" date 09/06/2016
bios0: LENOVO 20BSCTO1WW
acpi0 at bios0: ACPI 5.0
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP SLIC ASF! HPET ECDT APIC MCFG SSDT SSDT
SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT PCCT SSDT TCPA SSDT UEFI
MSDM BATB FPDT UEFI DMAR
acpi0: wakeup devices LID_(S4) SLPB(S3) IGBE(S4) EXP2(S4) XHCI(S3)
EHC1(S3)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 14318179 Hz
acpiec0 at acpi0
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz, 798.30 MHz, 06-3d-
04
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,
PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,D
TES64,MWAIT,DS-
CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE
4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAG
E1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI
1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,PT,MD_CLEAR,
IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz, 798.16 MHz, 06-3d-
04
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,
PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,D
TES64,MWAIT,DS-
CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE
4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAG
E1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI
1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,PT,MD_CLEAR,
IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 1, core 0, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz, 798.17 MHz, 06-3d-
04
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,
PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,D
TES64,MWAIT,DS-
CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE
4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAG
E1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI
1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,PT,MD_CLEAR,
IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 1, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz, 798.16 MHz, 06-3d-
04
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,
PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,D
TES64,MWAIT,DS-
CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE
4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAG
E1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI
1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,PT,MD_CLEAR,
IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 1, core 1, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec0, version 20, 40 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0xf800, bus 0-63
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (PEG_)
acpiprt2 at acpi0: bus 3 (EXP1)
acpiprt3 at acpi0: bus 4 (EXP2)
acpiprt4 at acpi0: bus -1 (EXP3)
acpiprt5 at acpi0: bus -1 (EXP6)
acpicpu0 at acpi0: C3(200@233 mwait.1@0x40), C2(200@148
mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpicpu1 at acpi0: C3(200@233 mwait.1@0x40), C2(200@148
mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpicpu2 at acpi0: C3(200@233 mwait.1@0x40), C2(200@148
mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpicpu3 at acpi0: C3(200@233 mwait.1@0x40), C2(200@148
mwait.1@0x33), C1(1000@1 

Re: Installation Media Self Integrity Check

2020-08-14 Thread Jordan Geoghegan




On 2020-08-12 21:30, Dan Peretz wrote:

Hello, the FAQ states this:
"The installXX.iso and installXX.fs images do not contain an
SHA256.sig file, so the installer will complain that it can't check
the signature of the included sets [...] This is because it would make
no sense for the installer to verify them. If someone were to make a
rogue installation image, they could certainly change the installer to
say the files were legitimate."
Although that's true for intentional modifications, it would still be
useful to have the installation medium perform a self integrity check
against accidental or natural data corruption. For example, Ubuntu
recently enabled a by-default integrity check, starting with release
20.04:
"Ubuntu now defaults to checking the integrity of the medium in use
when booting into live sessions. This can be skipped by hitting
Ctrl-C. We’ve enabled this because failed installs due to corrupt
downloads of installation media is one of the most common error
conditions that users encounter." (Source:
)
I would like to have OpenBSD include at least an unsigned SHA256 file
in the discs. The installer would then detect that the checksums are
unsigned and warn about the security implications, but it would let
the user run the check. I think it would be wise to make it check the
bsd.rd image that's actually booted when booting from the disc, and
not just the bsd.rd file set. (I get that the OpenBSD installer is
just a multipurpose "bsd.rd" RAM disk that can be found not just in
the installation discs, right?)

Thank you!



I think the whole idea is to check the ISO itself with signify before 
you even boot/write it. If you're concerned about whether the ISO was 
written to your install media correctly (or its integrity after the 
fact), then it should be pretty straightforward to check if the hashes 
match up. Once you've verified the SHA256 file from the mirrors, you 
know the hashes are valid (and if you're worried about MITM/compromised 
web hosts, the signify keys are published all over social media and 
blogs etc). Signify does its best to cope with the chicken and the egg 
problem (and everything that comes with trusting trust), but at the end 
of the day, you have to draw the line somewhere.


If you're spooked about the SHA256.sig warning, then you can install 
over the network from a mirror, but ultimately it's irrelevant. Check 
your ISO with signify and you're good to go.




Re: Adding more syspatch platform.

2020-08-13 Thread Jordan Geoghegan




On 2020-08-13 02:39, Sebastian Benoit wrote:

Jordan Geoghegan(jor...@geoghegan.ca) on 2020.08.12 10:32:21 -0700:


On 2020-08-12 02:08, Stuart Henderson wrote:

The only proxy we have for "what is really used" is dmesg submissions.
Since 6.7 release:

amd64   62
i3865
arm64   3
macppc  2
octeon  1

Based on this there isn't a great case for adding any more.


I didn't realize you guys used dmesg@ as a popularity gauge, I thought

s/popularity/usage/


it was just for sending dmesgs for interesting/new hardware. I figured
with something like Edgerouters with their standardized hardware that
repeat dmesgs would just serve to irritate the devs. I personally
administer more OpenBSD 6.x machines than are on that list you sent. I
can start hammering dmesg@, but then I'm gonna skew your stats and

In a way, you also skew the stats if you dont send dmesgs. And "you" means
everyone. Do it once every 6 months (for every release) i'd say.


you're gonna think that your userbase consists of a bunch of autists
that unironically run macppc, sparc64 and octeon in production
everywhere. In the small Canadian town I live in, I've got a big chunk
of it running on OpenBSD. I've got a bunch of old 90's / 2000's i386
stuff too, so I can send dmesgs in for that too, the reason I didn't is
I figured the hardware was already 'been there, done that'.

If the dmesgs dont show that certain hardware is still used, the drivers for
it will get deleted eventually. Another reason to show what you use.


As I mentioned earlier in the thread, I can afford to donate 2 octeon
machines to any devs that are interested (including shipping world wide;
any devs reading: please contact me if you're interested), and am also
happy to help out with octeon stuff in any way I can. Obviously you guys
aren't going to trust me to do anything syspatch related as I'm not a
dev, but I'd at least like it to be known that there are people who care
about the octeon port and who are willing make an effort for it.

Regards,

Jordan




Okay, thanks for the info Sebastian, I'll start doing twice annual dumps 
to dmesg@ for all my machines.


Regards,

Jordan



Re: Adding more syspatch platform.

2020-08-12 Thread Jordan Geoghegan




On 2020-08-12 02:08, Stuart Henderson wrote:

The only proxy we have for "what is really used" is dmesg submissions.
Since 6.7 release:

amd64   62
i3865
arm64   3
macppc  2
octeon  1

Based on this there isn't a great case for adding any more.



I didn't realize you guys used dmesg@ as a popularity gauge, I thought 
it was just for sending dmesgs for interesting/new hardware. I figured 
with something like Edgerouters with their standardized hardware that 
repeat dmesgs would just serve to irritate the devs. I personally 
administer more OpenBSD 6.x machines than are on that list you sent. I 
can start hammering dmesg@, but then I'm gonna skew your stats and 
you're gonna think that your userbase consists of a bunch of autists 
that unironically run macppc, sparc64 and octeon in production 
everywhere. In the small Canadian town I live in, I've got a big chunk 
of it running on OpenBSD. I've got a bunch of old 90's / 2000's i386 
stuff too, so I can send dmesgs in for that too, the reason I didn't is 
I figured the hardware was already 'been there, done that'.


As I mentioned earlier in the thread, I can afford to donate 2 octeon 
machines to any devs that are interested (including shipping world wide; 
any devs reading: please contact me if you're interested), and am also 
happy to help out with octeon stuff in any way I can. Obviously you guys 
aren't going to trust me to do anything syspatch related as I'm not a 
dev, but I'd at least like it to be known that there are people who care 
about the octeon port and who are willing make an effort for it.


Regards,

Jordan



Re: How many IPs can I block before taking a performance hit?

2020-08-12 Thread Jordan Geoghegan




On 2020-08-12 05:11, Alan McKay wrote:

Hey folks,

This is one that is difficult to test in a test environment.

I've got OpenBSD 6.5 on a relatively new pair of servers each with 8G RAM.

With some scripting I'm looking at feeding block IPs to the firewalls
to block bad-guys in near real time, but in theory if we got attacked
by a bot net or something like that, it could result in a few thousand
IPs being blocked.  Possibly even 10s of thousands.

Are there any real-world data out there on how big of a block list we
can handle without impacting performance?

We're doing the standard /etc/blacklist to load a table and then have
a block on the table right at the top of the ruleset.

thanks,
-Alan




At Otto said, if you're using tables, then you should be fine. I'm doing 
geoip blocking and all sorts of filtering using a pf table that contains 
over 200 undecillion addresses (that obviously includes CIDR block 
expansion):


# Entries (+-)
9482 addresses added.
10859 addresses deleted.

# Entries (expanded CIDR blocks)
IPv4 addresses in table:  966545967
IPv6 addresses in table:  298179424470603435988810818668701155328

fw$ wc -l < /etc/pf-badhost.txt
  146541




Re: Adding more syspatch platform.

2020-08-11 Thread Jordan Geoghegan




On 2020-08-11 20:35, Theo de Raadt wrote:

Jordan Geoghegan  wrote:


On 2020-08-11 15:50, Theo de Raadt wrote:

Predrag Punosevac  wrote:


Theo de Raadt  wrote:


No, it is a question of which additional platform, you avoided that
didn't you


octeon is the only one I can think of.

read below:


if you name one that less than 100 people use, then well come on

Not that it matters, but I have well over 100 users of OpenBSD on
octeon for the different client sites I manage. I'd love to see
syspatches for octeon, but I have no business telling you guys what to
do.

The people involved in building the syspatches don't even have octeons.



I'd be happy to donate a couple of octeon machines if access to hardware 
is an issue. I'm just a random internet person, so I imagine you guys 
wouldn't trust me to do the work to build the syspatches, but please do 
let me know if some octeon machines would be useful to you guys. I'd 
happily do the legwork myself but there doesn't appear to be a way to 
manually syspatch (please correct me if I'm wrong).


Regards,

Jordan



Re: Adding more syspatch platform.

2020-08-11 Thread Jordan Geoghegan




On 2020-08-11 15:50, Theo de Raadt wrote:

Predrag Punosevac  wrote:


Theo de Raadt  wrote:


No, it is a question of which additional platform, you avoided that
didn't you


octeon is the only one I can think of.

read below:


if you name one that less than 100 people use, then well come on


Not that it matters, but I have well over 100 users of OpenBSD on octeon 
for the different client sites I manage. I'd love to see syspatches for 
octeon, but I have no business telling you guys what to do.


Regards,

Jordan



Re: Way to find most active IPs for rate limiting with pf

2020-08-06 Thread Jordan Geoghegan




On 2020-08-06 13:46, Alan McKay wrote:

So I want to implement rate limiting, and to determine a reasonable
rate based on current traffic patterns I'd like to be able to figure
out which source IPs are generating the most connections and at what
rate.

Is there a way to do that?




There is likely a better way to do it, but a quick and dirty method to 
check for number of states for an IP address would be something like:


 $  doas pfctl -s state | grep -Fc "192.0.2.4"

or if you wanted to check for the number of tcp or udp states specifically:

  $  doas pfctl -s state | grep "^all tcp" | grep -Fc "192.0.2.4"


Regards,

Jordan



Re: CPU usage of httpd+slowcgi

2020-07-27 Thread Jordan Geoghegan




On 2020-07-24 03:16, Kihaguru Gathura wrote:

Hi,

Which of the following legacy CPU types is best suited for very busy web
server httpd+slowcgi

Niagara CPU Such as T2 - More parallel Threads and Low power per single
thread
Sparc64 CPU such as VI, VII - Fewer threads but more computing power per
thread.

How is multithreading utilization of httpd+slowcgi like?

Kind regards,

Kihaguru.


Hi  Kihaguru,

As with any computer, newer tends to be better with Moore's Law and all 
that. On sparc64 most of the logical cores that are shown are really 
just SMT pretending to be a bunch of cores. I have one machine that 
claims 128 cores, but in reality, its just 16 cores with 8-way SMT. 
sparc64 isn't renowned for its single core execution speed, so the 
faster the better in that regard.


In my experience with running OpenBSD on sparc64, the kernel biglock or 
crypto became a bottleneck before other things did. (I've used T3 and T4 
machines fairly extensively with OpenBSD). I've found that disk 
activity, networking and/or TLS would bottleneck before httpd became a 
bottleneck when I was running sparc64 web servers in production. If you 
are running very heavy scripts/programs with slowcgi, then you're 
results may be different.


Things have likely improved dramatically in the past year or two with 
all the work done on removing the biglock, but the moral of the story 
remains, fewer, faster cores are likely to produce superior performance 
to numerous low power cores.


Regards,

Jordan



Re: video capture / streaming

2020-07-25 Thread Jordan Geoghegan




On 2020-07-24 08:06, Rudolf Sykora wrote:

Dear list,


I'd like to stream video from my (usb-connected) camera via a web
server. The 1st I tried was to see if the camera works. So:

odin$ video
video: /dev/video: Permission denied
odin$ doas video
No protocol specified
video: cannot open display :0.0
odin$ xhost +
access control disabled, clients can connect from any host
odin$ doas video
^Codin$

Ie, at last I saw an X window with the video. But is it possible to run
the video command as an ordinary user? (I had to doas...)

Next I want to stream the video via some kind of a server.
On the internet I saw people using 'ffmpeg' to serve the file somehow,
as well as using 'nginx' with the 'rtmp' module. Is any of this the way
how you would achieve the goal, or is it possible to use, say, the 'httpd'
server?

At this moment I know very little both about streaming and available
servers to do the job, and that's why I am asking for recommendation
based on some experience and knowledge.

Thank you for any comments.


Ruda


Hi Ruda,

You can configure FFmpeg to dump an HLS stream into a location served by 
httpd. FFmpeg does support some USB cameras using the V4L2 protocol, but 
you may need to play around with formats and resolutions to get the 
optimal experience/performance. I've done this extensively, as I'm 
currently working on a CCTV project that will run natively on OpenBSD.


Regards,

Jordan



Re: Potential grep bug?

2020-07-11 Thread Jordan Geoghegan

Hi Demi,

On 2020-07-10 22:42, Demi M. Obenour wrote:

On 2020-06-23 22:29, Jordan Geoghegan wrote:

Hello,

I was working on a couple POSIX regular expressions to search for and validate 
IPv4 and IPv6 addresses with optional CIDR blocks, and encountered some strange 
behaviour from the base system grep.

I wanted to validate my regex against a list of every valid IPv4 address, so I 
generated a list with a zsh 1 liner:

  for i in {0..255}; do; echo $i.{0..255}.{0..255}.{0..255} ; done | tr 
'[:space:]' '\n' > IPv4.txt

My intentions were to test the regex by running it with 'grep -c' to confirm 
there was indeed 2^32 addresses matched, and I also wanted to benchmark and 
compare performance between BSD grep, GNU grep and ripgrep. The command I used:

    grep -Eoc 
"((25[0-5]|(2[0-4]|1{0,1}[[:digit:]]){0,1}[[:digit:]])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[[:digit:]]){0,1}[[:digit:]])(/[1-9]|/[1-2][[:digit:]]|/3[0-2])?"

My findings were surprising. Both GNU grep and ripgrep were able get through 
the file in roughly 10 and 20 minutes respectively, whereas the base system 
grep took over 20 hours! What interested me the most was that the base system 
grep when run with '-c' returned '0' for match count. It seems that 'grep -c' 
will have its counter overflow if there are more than 2^32-1 matches 
(4294967295) and then the counter will start counting from zero again for 
further matches.

Does OpenBSD use an NFA/DFA regular expression implementation, or a
backtracking one?  If it uses the latter, then your regex is probably
causing catastrophic backtracking.


Regards,

Jordan

Sincerely,

Demi



Ya you're probably right, I know GNU grep does a whole bunch of fancy 
optimizations that aren't done in BSD grep, the old GNU grep maintainer 
did a small write up on the FreeBSD mailing list about some of their tricks:

https://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html

My IPv6 regex causes grep to randomly segfault on MacOS (which uses an 
older version of the BSD/FreeGrep that OpenBSD uses I believe), so I 
imagine there's something going on under the hood with these regexes 
making grep thrash hard.



Regards,

Jordan



PF-BadHost Patch

2020-07-11 Thread Jordan Geoghegan

Hi everybody,

I don't want to spam the list here, but I figured this was important 
enough to warrant an announcement considering several hundred people 
downloaded the script last week.


tl;dr: Bugs found, patch your stuff

A couple bugs have been reported by users, one in RipGrep and another in 
find(1).
The RipGrep bug is a bit of a show-stopper, as it was discovered that 
its behaviour differed from that of grep(1) and GNU grep, where with the 
'-o' flag, it does not select the most exact match when given multiple 
search patterns ala 'pattern1|pattern2' it rather will select the first 
specified pattern that happens to match. What makes this significant is 
that the way this played out was RipGrep was only pulling the first 
digit in the CIDR block, ie '192.0.2.0/24' was instead matched as 
'192.0.2.0/2' . Big yikes. Mea culpa, should of caught that before I 
tried to be a tough guy and recommend an alternative to tried and true 
grep.


The find(1) bug has been quite persnickety to debug, as users report 
intermittent failures from find(1) where it pukes up error: 'find: 
-exec: cannot open "." '
I haven't been able to reproduce the error, so until I (or someone more 
knowledgeable) can figure out what's going on with that, that's going to 
have to be worked around.


I've released a patch to address both issues, you can find full 
instructions on the website: geoghegan.ca/pfbadhost.html


Quick start:

$ ftp https://geoghegan.ca/pub/pf-badhost/0.4/patches/pf-badhost_p0.patch
# patch Hey folks, just thought I'd share with you that I've released the 
latest versions of pf-badhost and unbound-adblock.


pf-badhost webpage: https://www.geoghegan.ca/pfbadhost.html
unbound-adblock webage: https://www.geoghegan.ca/unbound-adblock.html

Key pf-badhost changes:

* pf-badhost goes portable, we now support 
{Open,Free,Net,Dragonfly}BSD as well as MacOS!
* Support for IPv6 subnet aggregation added thanks to the excellent 
aggregate6 utility written by job@

* Greatly improved IPv6 handling in general
* User configuration section added for configuring whitelists and 
custom blocklists

* Bogon filtering added
* Greatly improved error handling


Key unbound-adblock changes:

* unbound-adblock goes portable, we now support 
{Open,Free,Net,Dragonfly}BSD as well as Linux!

* Greatly improved error handling and input sanitation
* User configuration section added for configuring whitelists and 
custom blocklists


pf-badhost changelog: 
https://www.geoghegan.ca/pub/pf-badhost/0.4/changelog.txt
unbound-adblock changelog: 
https://www.geoghegan.ca/pub/unbound-adblock/0.4/changelog.txt






Re: sysupgrade failure due to boot.conf

2020-07-10 Thread Jordan Geoghegan




On 2020-07-10 15:37, Alfred Morgan wrote:

Please, I have had this problem for several versions now and it still isn't
working right.
I have this on all three of my servers:
echo boot > /etc/boot.conf

I have this boot.conf because openbsd fails to boot (on all three servers)
because it hangs on the boot> prompt because of some ghost input (I have no
keyboard plugged in). So I was told that putting "boot" in my boot.conf
would solve the problem and it did the trick. BUT, sysupgrade now fails
trying to upgrade 6.6 -> 6.7. So what can I put in my boot.conf that will
ignore the ghost input in the boot> prompt and allow sysupgrade to succeed?

I feel that it's a bug in sysupgrade that it doesn't behave the same having
"boot" in the boot.conf. Any help?

-alfred



Relevant xkcd? https://xkcd.com/1172/



Re: Unbound Configuration

2020-07-10 Thread Jordan Geoghegan




On 2020-07-10 14:29, ken.hendrick...@l3harris.com wrote:

--- I asked:

What I would like to do now is make the *simplest
possible* unbound.conf file and get it working.

Thinking that an absolutely empty unbound.conf file
would be the simplest, I tried it.  It doesn't work.

Can anybody help me with the simplest possible
unbound.conf file???

Thanks,
Ken


   


CONFIDENTIALITY NOTICE: This email and any attachments are for the sole use of 
the intended recipient and may contain material that is proprietary, 
confidential, privileged or otherwise legally protected or restricted under 
applicable government laws. Any review, disclosure, distributing or other use 
without expressed permission of the sender is strictly prohibited. If you are 
not the intended recipient, please contact the sender and delete all copies 
without reading, printing, or saving.




This FAQ page should be a useful starting point:

https://www.openbsd.org/faq/pf/example1.html#dns



pf-badhost + unbound adblock v4 released

2020-07-01 Thread Jordan Geoghegan
Hey folks, just thought I'd share with you that I've released the latest 
versions of pf-badhost and unbound-adblock.


pf-badhost webpage: https://www.geoghegan.ca/pfbadhost.html
unbound-adblock webage: https://www.geoghegan.ca/unbound-adblock.html

Key pf-badhost changes:

* pf-badhost goes portable, we now support {Open,Free,Net,Dragonfly}BSD 
as well as MacOS!
* Support for IPv6 subnet aggregation added thanks to the excellent 
aggregate6 utility written by job@

* Greatly improved IPv6 handling in general
* User configuration section added for configuring whitelists and custom 
blocklists

* Bogon filtering added
* Greatly improved error handling


Key unbound-adblock changes:

* unbound-adblock goes portable, we now support 
{Open,Free,Net,Dragonfly}BSD as well as Linux!

* Greatly improved error handling and input sanitation
* User configuration section added for configuring whitelists and custom 
blocklists


pf-badhost changelog: 
https://www.geoghegan.ca/pub/pf-badhost/0.4/changelog.txt
unbound-adblock changelog: 
https://www.geoghegan.ca/pub/unbound-adblock/0.4/changelog.txt




Re: How do I get the man page for a package I haven't installed yet?

2020-06-26 Thread Jordan Geoghegan




On 2020-06-26 20:03, Theo de Raadt wrote:

Jordan Geoghegan  wrote:


On 2020-06-26 18:45, Theo de Raadt wrote:

Jordan Geoghegan  wrote:



On 2020-06-26 13:43, Marc Espie wrote:

On Tue, Jun 23, 2020 at 12:20:35PM -0600, Theo de Raadt wrote:

Ottavio Caruso  wrote:


Hi,

Unless I've got it all wrong, <https://man.openbsd.org/> will only
display man pages for programs and commands in base. Is there a way to
display the man page for a package/port I haven't installed and/or
downloaded yet? (This assumes I haven't downloaded the ports cvs
tree).

Doing that would be very annoying and painful, and very few people
would want it.  It would also substantially degrade the clarity at
man.openbsd.org

Actually, it ought to be feasible to have the same mechanism in place for
base  as a third party mechanism.

I don't think it would be that difficult to setup, this obviously ought to
be separate from the main OpenBSD installation, as the quality of manpages
from ports is often not up-to-par compared to base.

Both Ingo and naddy and I, we've been routinely passing all manpages from
all packages through groff and mandoc and makewhatis to the point that
over 99% of them would be clean for a usage similar to man.openbsd.org


FreeBSD appears to offer manual pages from ports on their man page
website: https://www.freebsd.org/cgi/man.cgi

Not advocating for anything, just thought I'd point it out.

Completely irrelevant.


I thought it was relevant for folks looking for http access to ports
manpages, as the FreeBSD and OpenBSD ports trees overlap
significantly. I often use that site when I'm on a machine that
doesn't happen to have the particular package installed whose manpage
I want to view.

It is very easy for outsiders to ask a project to do more, MORE MORE
MORE, and not understand there are a limited number of people doing the
work.

So if this gets done, something else will not get done, or will get done
less well.

And it will be your fault.




I wasn't asking for anything, I was just trying to be helpful and share 
a resource I've personally found useful. I don't feel strongly about any 
of this, so consider the conversation over.


Regards,

Jordan



Re: How do I get the man page for a package I haven't installed yet?

2020-06-26 Thread Jordan Geoghegan




On 2020-06-26 18:45, Theo de Raadt wrote:

Jordan Geoghegan  wrote:




On 2020-06-26 13:43, Marc Espie wrote:

On Tue, Jun 23, 2020 at 12:20:35PM -0600, Theo de Raadt wrote:

Ottavio Caruso  wrote:


Hi,

Unless I've got it all wrong, <https://man.openbsd.org/> will only
display man pages for programs and commands in base. Is there a way to
display the man page for a package/port I haven't installed and/or
downloaded yet? (This assumes I haven't downloaded the ports cvs
tree).

Doing that would be very annoying and painful, and very few people
would want it.  It would also substantially degrade the clarity at
man.openbsd.org

Actually, it ought to be feasible to have the same mechanism in place for
base  as a third party mechanism.

I don't think it would be that difficult to setup, this obviously ought to
be separate from the main OpenBSD installation, as the quality of manpages
from ports is often not up-to-par compared to base.

Both Ingo and naddy and I, we've been routinely passing all manpages from
all packages through groff and mandoc and makewhatis to the point that
over 99% of them would be clean for a usage similar to man.openbsd.org


FreeBSD appears to offer manual pages from ports on their man page
website: https://www.freebsd.org/cgi/man.cgi

Not advocating for anything, just thought I'd point it out.

Completely irrelevant.



I thought it was relevant for folks looking for http access to ports 
manpages, as the FreeBSD and OpenBSD ports trees overlap significantly. 
I often use that site when I'm on a machine that doesn't happen to have 
the particular package installed whose manpage I want to view.


Regards,

Jordan



Re: How do I get the man page for a package I haven't installed yet?

2020-06-26 Thread Jordan Geoghegan




On 2020-06-26 13:43, Marc Espie wrote:

On Tue, Jun 23, 2020 at 12:20:35PM -0600, Theo de Raadt wrote:

Ottavio Caruso  wrote:


Hi,

Unless I've got it all wrong,  will only
display man pages for programs and commands in base. Is there a way to
display the man page for a package/port I haven't installed and/or
downloaded yet? (This assumes I haven't downloaded the ports cvs
tree).

Doing that would be very annoying and painful, and very few people
would want it.  It would also substantially degrade the clarity at
man.openbsd.org

Actually, it ought to be feasible to have the same mechanism in place for
base  as a third party mechanism.

I don't think it would be that difficult to setup, this obviously ought to
be separate from the main OpenBSD installation, as the quality of manpages
from ports is often not up-to-par compared to base.

Both Ingo and naddy and I, we've been routinely passing all manpages from
all packages through groff and mandoc and makewhatis to the point that
over 99% of them would be clean for a usage similar to man.openbsd.org



FreeBSD appears to offer manual pages from ports on their man page 
website: https://www.freebsd.org/cgi/man.cgi


Not advocating for anything, just thought I'd point it out.



Potential grep bug?

2020-06-23 Thread Jordan Geoghegan

Hello,

I was working on a couple POSIX regular expressions to search for and 
validate IPv4 and IPv6 addresses with optional CIDR blocks, and 
encountered some strange behaviour from the base system grep.


I wanted to validate my regex against a list of every valid IPv4 
address, so I generated a list with a zsh 1 liner:


 for i in {0..255}; do; echo $i.{0..255}.{0..255}.{0..255} ; done | 
tr '[:space:]' '\n' > IPv4.txt


My intentions were to test the regex by running it with 'grep -c' to 
confirm there was indeed 2^32 addresses matched, and I also wanted to 
benchmark and compare performance between BSD grep, GNU grep and 
ripgrep. The command I used:


   grep -Eoc 
"((25[0-5]|(2[0-4]|1{0,1}[[:digit:]]){0,1}[[:digit:]])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[[:digit:]]){0,1}[[:digit:]])(/[1-9]|/[1-2][[:digit:]]|/3[0-2])?"


My findings were surprising. Both GNU grep and ripgrep were able get 
through the file in roughly 10 and 20 minutes respectively, whereas the 
base system grep took over 20 hours! What interested me the most was 
that the base system grep when run with '-c' returned '0' for match 
count. It seems that 'grep -c' will have its counter overflow if there 
are more than 2^32-1 matches (4294967295) and then the counter will 
start counting from zero again for further matches.


    ryzen$ time zcat IPv4.txt.gz | grep -Eoc "((25[0-5]|(2[0-4]|1{0,1}...
    0
    1222m09.32s real  1224m28.02s user 1m16.17s system

    ryzen$ time zcat allip.txt.gz | ggrep -Eoc "((25[0-5]|(2[0-4]|1{0,1}...
    4294967296
    10m00.38s real    11m40.57s user 0m30.55s system

    ryzen$ time rg -zoc "((25[0-5]|(2[0-4]|1{0,1}...
    4294967296
    21m06.36s real    27m06.04s user 0m50.08s system

# See the counter overflow/reset:
    jot 4294967350 | grep -c "^[[:digit:]]"
    54

All testing was done on a Ryzen desktop machine running 6.7 stable.

The grep counting bug can be reproduced with this command:
   jot 4294967296 | nice grep -c "^[[:digit:]]"

Regards,

Jordan



Re: Any idea/suggestion for old Cisco router to be use running OpenBSD current for WG?

2020-06-23 Thread Jordan Geoghegan
I don't know much about Cisco hardware, but I've had great luck with the 
Edgerouter line of products. I've run my home network on an Edgerouter 
Pro for several years now without issue, and have dozens of ER4 and 
ER-Lite devices out in the wild.


If you're looking for non-x86 routing solutions, then the Edgerouter is 
one of the best bets.


Regards,

Jordan

On 2020-06-23 09:01, Daniel Ouellet wrote:

Hi,

This might be a bit weird question, but I saw the wireguard being put in
the kernel in the last few days and I am very existed abut it oppose to
use the package on it and even today there was more on it.

Many thanks for this!!!

I also know there was effort and some Cisco router can run OpenBSD very
well, however I have no clue as to any of this stand now.

I don't have a problem to use APU type or other Ubiquit for small
OpenBSD router, but I wonder about using Cisco instead. The only reason
is for may be more stability, most likely less performance for sure, but
less change to have corrupted reboot on power lost, etc.

And sadly for some customers having what they see as computer as router
don't make them fell good, but seeing a Cisco box kind of wipe out the
impression. I am not saying it's justify, but perception is sometime
everything, but if I have my say in it I want all my routers to be
OpenBSD as much as I can where the needs is not to multiple Gb in speed.

So, any suggestion or updates as to what's now available and hopefully
in use now.

I really don't care for any special model, or even Juniper, as long as I
can put OpenBSD on it.

So any feedback as to where it's stand now and what's usable in a
reliable way would be greatly appreciated.

And yes I know I may well get better performance in some cases with a
small APU device then a Cisco one, but that's for what we all know may
not be logical to be used, but for sadly how some clients may fell, not
knowing any better.

I guess you can see that as some people do security by obstruction, but
we al know it's not more secure, this is routing by obstruction I guess
and may be less performant, but achieve comfort obstruction confidence.

I just have no clue if wireguard needs to be run, what can be achieve as
the CPU in all Cisco device is always under power, we all know that.

This may not go anywhere, however I liked to look even if for nothing
else then just being fun to do if that can't even be usable.

Many thanks for your time and feedback.

Daniel

PS; And yes, that's most likely stupid I know. Sometime what's used is
not always what make sense for other reason that are stupid.





Re: Potential awk bug?

2020-06-13 Thread Jordan Geoghegan




On 2020-06-13 05:14, Stuart Henderson wrote:

On 2020-06-12, Jordan Geoghegan  wrote:


On 2020-06-08 11:29, Todd C. Miller wrote:

On Sun, 07 Jun 2020 17:02:03 -0700, Jordan Geoghegan wrote:


Thanks for the quick response. I certainly wasn't expecting to find an
ancient bug like this. Should I be reporting this bug upstream, or are
you planning on upstreaming a diff?

I've created a pull request to fix this upstream:
  https://github.com/onetrueawk/awk/pull/80

   - todd

Sorry for the late reply, I've been AFK for the last week.

Excited to see base system awk get synced with upstream. Kudos to you
and Philip for such a swift and effective response.

btw, note that it's not a direct copy of upstream, OpenBSD's version has
various changes (at least pledge and use of bounded string functions).




Thanks for the info Stuart,  that's awesome, I didn't realise awk was 
pledged. I appreciate the knowledge tidbits you drop from time to time. 
Your bit the other day about having ftp(1) output to stdout to drop all 
filesystem privileges blew my mind and I've subsequently updated most of 
my scripts to take advantage of it.


Regards,

Jordan Geoghegan



Re: Potential awk bug?

2020-06-12 Thread Jordan Geoghegan




On 2020-06-08 11:29, Todd C. Miller wrote:

On Sun, 07 Jun 2020 17:02:03 -0700, Jordan Geoghegan wrote:


Thanks for the quick response. I certainly wasn't expecting to find an
ancient bug like this. Should I be reporting this bug upstream, or are
you planning on upstreaming a diff?

I've created a pull request to fix this upstream:
 https://github.com/onetrueawk/awk/pull/80

  - todd


Sorry for the late reply, I've been AFK for the last week.

Excited to see base system awk get synced with upstream. Kudos to you 
and Philip for such a swift and effective response.


Regards,

Jordan Geoghegan



Re: OpenBSD alternatives to Pi-Hole

2020-06-12 Thread Jordan Geoghegan




On 2020-06-12 14:01, George wrote:


On 2020-06-12 3:41 p.m., Maurice McCarthy wrote:
You could have a look at 
https://www.geoghegan.ca/unbound-adblock.html and

https://www.geoghegan.ca/pfbadhost.html


Simply great! Will definitely try these out.

Merci!

George



Hey there,

I'm the author of those scripts. In response to concerns about 
heavyness/memory use of DNS blocklists:  unbound-adblock is pretty light 
on memory (~30MB of RAM usage) as we serve NXDOMAIN responses instead of 
redirecting to 0.0.0.0 etc. By doing this we save a massive amount of 
memory that would otherwise be spent mapping each domain to a black hole 
address. I run unbound-adblock on many Edgerouter Lites and havent had 
any issues.


Regards,

Jordan Geoghegan



Re: Potential awk bug?

2020-06-07 Thread Jordan Geoghegan

Hi Philip,

Thanks for the quick response. I certainly wasn't expecting to find an 
ancient bug like this. Should I be reporting this bug upstream, or are 
you planning on upstreaming a diff?


Regards,

Jordan



On 2020-06-06 20:16, Philip Guenther wrote:
On Sat, Jun 6, 2020 at 5:08 PM Zé Loff <mailto:zel...@zeloff.org>> wrote:


On Sat, Jun 06, 2020 at 03:51:58PM -0700, Jordan Geoghegan wrote:
> I'm working on a simple awk snippet to convert the IP range data
listed in
> the Extended Delegation Statistics data from ARIN [1] and
convert it into
> CIDR blocks. I have a snippet that works perfectly fine on mawk
and gawk,
> but not on the base system awk. I'm 99% sure I'm not using any
GNUisms, as
> when I break the command up into two parts, it works perfectly.
>
> The snippet below does not work with base awk, but does work
with gawk and
> mawk: (Running on 6.6 -stable system)
>
>   awk -F '|' '{ if ( $3 == "ipv4" && $2 == "US")
printf("%s/%d\n", $4,
> 32-log($5)/log(2))}' delegated-arin-extended-latest.txt
>
>
> The command does output data, but it also throws errors for
certain lines:
>
>   awk: log result out of range
>   input record number 94027, file delegated-arin-extended-latest.txt
>   source line number 1
>
> Most CIDR blocks are calculated correctly, but about 10% of them
have errors
> (ie something that should calculated to be a /24 is instead
calculated to be
> a /30).

...

I have no idea about what is going on, but FWIW I can reproduce
this on
i386 6.7-stable and amd64 6.7-current (well, current-ish, #232).
Truncating the file to a single offending line produces the same
result:
log($5) is out of range.

It appears to have something to do with the last field. Removing it or
changing some of its characters seems to work, e.g.:


arin|US|ipv4|216.250.144.0|4096|20050503|allocated|5e58386636aa775c2106140445cf2c30

arin|US|ipv4|216.250.144.0|4096|20050503|allocated|5a58386636aa775c2106140445cf2c30
                                                    ^
Fails on the first line but works on the second.


Hah!  Nice observation!

The last field of the first line looks kinda like a number in 
scientific notation, but when awk internally tries to set up the 
fields it generates an ERANGE error...and the global errno variable is 
left with that value.  Several builtins in awk, including log(), 
perform operations and then check whether errno is set to EDOM or 
ERANGE but fail to clear errno beforehand.


The fix is to zero errno before all the code sequences that use the 
errcheck() function, ala:


--- run.c       13 Aug 2019 10:45:56 -      1.44
+++ run.c       7 Jun 2020 03:14:38 -
@@ -26,6 +26,7 @@ THIS SOFTWARE.
 #define DEBUG
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
@@ -1041,8 +1042,10 @@ Cell *arith(Node **a, int n)     /* a[0] + a
        case POWER:
                if (j >= 0 && modf(j, ) == 0.0)       /* pos integer 
exponent */

                        i = ipow(i, (int) j);
-               else
+               else {
+                       errno = 0;
                        i = errcheck(pow(i, j), "pow");
+               }
                break;
        default:        /* can't happen */
                FATAL("illegal arithmetic operator %d", n);
@@ -1135,8 +1138,10 @@ Cell *assign(Node **a, int n)    /* a[0] =
        case POWEQ:
                if (yf >= 0 && modf(yf, ) == 0.0)     /* pos integer 
exponent */

                        xf = ipow(xf, (int) yf);
-               else
+               else {
+                       errno = 0;
                        xf = errcheck(pow(xf, yf), "pow");
+               }
                break;
        default:
                FATAL("illegal assignment operator %d", n);
@@ -1499,12 +1504,15 @@ Cell *bltin(Node **a, int n)    /* builtin
                        u = strlen(getsval(x));
                break;
        case FLOG:
+               errno = 0;
                u = errcheck(log(getfval(x)), "log"); break;
        case FINT:
                modf(getfval(x), ); break;
        case FEXP:
+               errno = 0;
                u = errcheck(exp(getfval(x)), "exp"); break;
        case FSQRT:
+               errno = 0;
                u = errcheck(sqrt(getfval(x)), "sqrt"); break;
        case FSIN:
                u = sin(getfval(x)); break;


Todd, are we up to date with upstream, or is this latent there too?


Philip Guenther





Potential awk bug?

2020-06-06 Thread Jordan Geoghegan

Hello,

I was hoping the fine folks here could give me a quick sanity check, I'm 
by no means an awk guru, so I'm likely missing something obvious. I 
wanted to ask here quickly before I started flapping my gums on bugs@.


I'm working on a simple awk snippet to convert the IP range data listed 
in the Extended Delegation Statistics data from ARIN [1] and convert it 
into CIDR blocks. I have a snippet that works perfectly fine on mawk and 
gawk, but not on the base system awk. I'm 99% sure I'm not using any 
GNUisms, as when I break the command up into two parts, it works perfectly.


The snippet below does not work with base awk, but does work with gawk 
and mawk: (Running on 6.6 -stable system)


  awk -F '|' '{ if ( $3 == "ipv4" && $2 == "US") printf("%s/%d\n", $4, 
32-log($5)/log(2))}' delegated-arin-extended-latest.txt



The command does output data, but it also throws errors for certain lines:

  awk: log result out of range
  input record number 94027, file delegated-arin-extended-latest.txt
  source line number 1

Most CIDR blocks are calculated correctly, but about 10% of them have 
errors (ie something that should calculated to be a /24 is instead 
calculated to be a /30).


However, when I break it up into two parts, it produces the expected output:

  awk -F '|' '{ if ( $3 == "ipv4" && $2 == "US") print($4, $5)}' 
delegated-arin-extended-latest.txt | awk  '{printf("%s/%d\n", $1, 
32-log($2)/log(2)) }'


As you can see, the same number of lines are printed, but the hashes are 
different.


  luna$ gawk -F '|' '{ if ( $3 == "ipv4" && $2 == "US") 
printf("%s/%d\n", $4, 32-log($5)/log(2))}' delegated-*-latest.txt | wc -l

     56446
  luna$ mawk -F '|' '{ if ( $3 == "ipv4" && $2 == "US") 
printf("%s/%d\n", $4, 32-log($5)/log(2))}' delegated-*-latest.txt | wc -l

     56446
  luna$ awk -F '|' '{ if ( $3 == "ipv4" && $2 == "US") 
printf("%s/%d\n", $4, 32-log($5)/log(2))}' delegated-*-latest.txt 
2>/dev/null | wc -l

     56446

  luna$ awk -F '|' '{ if ( $3 == "ipv4" && $2 == "US") 
printf("%s/%d\n", $4, 32-log($5)/log(2))}' 
delegated-arin-extended-latest.txt 2>/dev/null | md5

    6f549bbc0799bc202c12695f8530d1df
  luna$ gawk -F '|' '{ if ( $3 == "ipv4" && $2 == "US") 
printf("%s/%d\n", $4, 32-log($5)/log(2))}' 
delegated-arin-extended-latest.txt 2>/dev/null | md5

    40c28b8ebfd2796e1ae15d9f6401c0c1
  luna$ mawk -F '|' '{ if ( $3 == "ipv4" && $2 == "US") 
printf("%s/%d\n", $4, 32-log($5)/log(2))}' 
delegated-arin-extended-latest.txt 2>/dev/null | md5

    40c28b8ebfd2796e1ae15d9f6401c0c1


Example of the differences:

--- mawk.txt    Sat Jun  6 18:43:30 2020
+++ awk.txt Sat Jun  6 18:43:38 2020
@@ -29,7 +29,7 @@
 9.64.0.0/10
 9.128.0.0/9
 11.0.0.0/8
-12.0.0.0/8
+12.0.0.0/30
 13.0.0.0/11
 13.32.0.0/12
 13.48.0.0/14
@@ -415,7 +415,7 @@
 23.90.64.0/20
 23.90.80.0/21
 23.90.88.0/22
-23.90.92.0/22
+23.90.92.0/30
 23.90.96.0/19
 23.91.0.0/19
 23.91.32.0/19
@@ -545,8 +545,8 @@
 23.133.224.0/24
 23.133.240.0/24
 23.134.0.0/24
-23.134.16.0/24
-23.134.17.0/24
+23.134.16.0/30
+23.134.17.0/30


Any insight or advice would be much appreciated.

Regards,

Jordan

[1] https://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest




Re: Filling a 4TB Disk with Random Data

2020-06-01 Thread Jordan Geoghegan




On 2020-06-01 06:58, Justin Noor wrote:

Hi Misc,

Has anyone ever filled a 4TB disk with random data and/or zeros with
OpenBSD?

How long did it take? What did you use (dd, openssl)? Can you share the
command that you used?

Thank you so much



I've used OpenBSD to overwrite up to 8TB disks. I use a large block size 
with 'dd' and make sure to use /dev/rsdX (the 'r' makes things much 
faster).




Re: 10Gbps X520 network adapter only passing 3.5Gbps

2020-05-06 Thread Jordan Geoghegan




On 2020-05-06 04:04, Stuart Henderson wrote:

On 2020-05-06, Jordan Geoghegan  wrote:


On 2020-05-04 06:42, Kalle Kadakas wrote:

Greetings OpenBSD community,

I am running into severe bandwidth limitations whilst passing traffic
   through an OpenBSD firewall.
The NIC in use is an Intel 10Gb 2-port X520 adapter from which I would
   hope to pass through at least 7Gbps+, yet the best results I have
   gotten is only around 3.5Gbps.

The results of bandwidth measurements (iperf for 30sec...
   

As has been discussed on misc previously, iperf is not suitable for
benchmarking networking throughput on OpenBSD. It ends up just
benchmarking the gettimeofday syscall (something that is cheap on Linux,
but relatively expensive on OpenBSD I'm told).

clock_gettime. It's iperf3 that calls this often; iperf not so much.
But when testing with default options, you may see higher numbers from
iperf3: the direct comparison isn't fair though because it uses 128K
TCP buffers by default, whereas iperf uses the OS default.

On Linux clock_gettime often doesn't use a system call, on OpenBSD it
does (and with some of the mitigations for cpu bugs, system calls are
more expensive than they used to be).


For best results, use tcpbench for your OpenBSD networking benchmarks.

For accurate results of packet forwarding performance, use fast packet
sources/sinks running whatever OS either side of an OpenBSD router.




Thanks for clarifying Stuart, I knew I was about 70% of the way there.

Jordan



Re: 10Gbps X520 network adapter only passing 3.5Gbps

2020-05-06 Thread Jordan Geoghegan




On 2020-05-04 06:42, Kalle Kadakas wrote:

Greetings OpenBSD community,

I am running into severe bandwidth limitations whilst passing traffic
  through an OpenBSD firewall.
The NIC in use is an Intel 10Gb 2-port X520 adapter from which I would
  hope to pass through at least 7Gbps+, yet the best results I have
  gotten is only around 3.5Gbps.

The results of bandwidth measurements (iperf for 30sec...
  


As has been discussed on misc previously, iperf is not suitable for 
benchmarking networking throughput on OpenBSD. It ends up just 
benchmarking the gettimeofday syscall (something that is cheap on Linux, 
but relatively expensive on OpenBSD I'm told). For best results, use 
tcpbench for your OpenBSD networking benchmarks.





Re: SpeedTest-cli results accuracy ?

2020-05-05 Thread Jordan Geoghegan

Hi Kanto,

The Edgerouter Lite will not push much more than 200mbps, so that will 
certainly be a bottleneck. The only reason the ERlite can push 1Gbit 
with stock firmware is because of proprietary cut through routing and 
other garbage -- what they dont tell you is that as soon as you enable 
QoS or any fancy firewalling (or even IPv6!) you lose that hardware 
acceleration and all forwarding/packet handling is done purely on the CPU.


With regards to speedtest-cli, those results are not accurate. The only 
way to be sure it to use something like tcpbench and test it against 
multiple locations (I usually spin up a few vultr VPS in various 
locations to confirm my speeds for example). Also, I'm sure you already 
know this, but you also should never run the benchmarking program on 
your router, as that will obviously skew the results.


Cheers,

Jordan



On 2020-05-05 17:47, Kanto Andria wrote:

Hello all,
First post here. So please be indulgent ;-)). My question is about the 
speedtest-cli tool and the tests results with OpenBsd.Let me explain. I have 
multiple machines - physical and virtual - mix of BSD and Linux - and I am in a 
process of rebuilding my Firewall - obviously with OpenBSD/PF.
I have had an old Firewall using EdgeRouterLite which is broken now (no 
response from keyboard input using the console access - different story).With 
the ERL FW, when I increase  the ISP contract speeds from  200/30 Mbps to 
400/50 Mbps - doing a speed using any computer from the LAN did not pass over 
around ~220/35 Mbps.
The provider provided a Zyxel (if this matters) which "acts temporarily" as the 
Firewall + DHCP, etc. Any speedtest from Linux, Windows (son's game computer) got around 
the 400 Mbps/50Mbps or more.
The OpenBSD station (running 6.6) gets no more than 250 Mbps - the new Firewall 
I'm building shows the same results (see dmesg below) - no GUI for the 2 
machines - using speedtest-cli.
Following are the tests - machines - I ran with their respective results:
- Lenovo ThinkCenter - OpenBSD 6.6 - speedtest-cli : ~230/37 Mbps- Future 
Firewall (acting as workstation for the test)  - OpenBSD 6.6: around the same 
results- OpenBSD 6.6 running as VM on Manjaro Linux - around the same results
- Manjaro Linux (Physical) hosting the OBSD VM - reach around or more the 
~420/52 Mbps (using speedtest-cli and the Browser)- PFSense running as VM on 
Manjaro Linux - where I installed speedtest-cli - reach around the ~400/50 Mbps
Even they are not the same tools (iperf vs speedtest-cli) - running iperf3 
between OBSD vs OBSD/Linux and/or tcpbench, the tests display results close to 
960 Mbps.

So my question is can: I rely on the on the speedtest results? What else should 
I verify? Changing cables/direct connections to the current router were already 
done.

Thanks for any inputs and clarification.
Kanto

dmesg for the future Router/Firewall
OpenBSD 6.6 (GENERIC.MP) #8: Fri Apr 17 15:06:32 MDT 2020
     
r...@syspatch-66-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 8489873408 (8096MB)
avail mem = 8219873280 (7839MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xec410 (83 entries)
bios0: vendor American Megatrends Inc. version "5.6.5" date 06/30/2018
bios0: INTEL Corporation Q3XXG4-P
acpi0 at bios0: ACPI 5.0
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP APIC FPDT FIDT MCFG HPET SSDT UEFI SSDT ASF! SSDT SSDT 
SSDT DMAR
acpi0: wakeup devices PEG0(S4) PEGP(S4) PEG1(S4) PEGP(S4) PEG2(S4) PEGP(S4) 
RP01(S4) PXSX(S4) RP02(S4) PXSX(S4) RP03(S4) PXSX(S4) RP04(S4) PXSX(S4) 
RP05(S4) PXSX(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM) i3-4030U CPU @ 1.90GHz, 1895.92 MHz, 06-45-01
cpu0: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Core(TM) i3-4030U CPU @ 1.90GHz, 1895.62 MHz, 06-45-01
cpu1: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu1: 

Re: More than 16 partitions

2020-04-23 Thread Jordan Geoghegan




On 2020-04-23 11:45, zeurk...@volny.cz wrote:

"Jan Betlach"  wrote:

For a non-native English speaker like myself, it is very difficult to
read your mestuff...

Your point is well-taken (though this is just the way mespeaks); yet,
Theo is a native speaker, and he seems to have completely missed the
content of merecent responses.

Weird, isn't it? Anyway, as this would appear to be quite OT, me'd
suggest we continue this (if at all) in private mail...


Jan

Take care,

 --zeurkous.



Say Hi to Boss Nass for me next time you're in Otoh Gunga.

Jordan



Re: Openbsd mirrors in Kazakhstan?

2020-04-10 Thread Jordan Geoghegan




On 2020-04-10 10:11, Nikita Stepanov wrote:

Openbsd mirrors in Kazakhstan?


The Russian mirror is hosted by Yandex. Moscow is likely the closest 
mirror to Kazakhstan, otherwise I would recommend trying one of the CDN 
options




Re: Hosting a CDN question

2020-03-16 Thread Jordan Geoghegan




On 2020-03-16 03:26, Flipchan wrote:

Hey all,

My company needs to put up a cdn for fast hosting of javascript, images and css 
for websites, and then i would need something faster then httpd.


Does anyone here run a cdn for static website content?

If so what software did u use to set it up ?

have a good one
Sincerely
Filip


What level of traffic are you looking to push? Have you done any tests 
to confirm httpd was unsuitable or was the bottleneck?


When coupled with relayd, you can have a very powerful setup. I have a 
setup with relayd + httpd on modest hardware, and I can push over 5,000 
requests per second.




Re: Jitsi on OpenBSD

2020-03-16 Thread Jordan Geoghegan




On 2020-03-16 03:59, Edd Barrett wrote:

Hi,

(CC people who may be knowledgable in this area)

I was wondering if anyone has got the Jitsi (https://jitsi.org/)
web-client working on OpenBSD?

It's open-source (and self-hostable) video conferencing.

No prizes for guessing why I'm investigating this :P

I've just (quickly) tried the browser client in firefox:

  - It recognises my microphone and my camera.
  - Thumbnail shows local video feed OK.
  - I can hear audio from an android participant.
  - The android participant cannot hear the audio from the OpenBSD machine.
  - The video is super-flaky on both ends.

Did this, as per firefox README:

  - I have sysctl kern.audio.record=1.
  - I chowned /dev/video0 to me.

This evening I'm going to have a deeper play around (e.g. verify if mic
works in aucat), but if anyone has got this working before, I'd love to
hear what tweaks they had to do.

Could be that the jitsi server is overloaded.

Thanks

You may also want to try out talky.io, its treated my me well over the 
years and has a very similar concept to jitsi.


Jordan

(ps sorry for sending you this twice, forgot to cc misc on my first attempt)



Re: riscv

2020-03-15 Thread Jordan Geoghegan




On 2020-03-14 23:19, Mike Larkin wrote:

On Sat, Mar 14, 2020 at 11:18:11PM -0700, Mike Larkin wrote:

On Fri, Mar 13, 2020 at 02:12:19PM -0700, Jordan Geoghegan wrote:


On 2020-03-13 09:50, Christian Weisgerber wrote:

On 2020-03-13, "Peter J. Philipp"  wrote:


Any developer working on a riscv port and willing to share their unofficial
work for possible future collaboration?

I think I'd have heard by now if somebody was, so I'll go out on a
limb and say no, nobody's working on a RISC-V port.


I stumbled across this a while back, this guy at least claims to be
attempting a port to RISC-V...

https://github.com/MengshiLi/openbsd-riscv-notes


We have a riscv64 kernel booting up to the rootdev prompt, and are working on

PS, "We" here is my student team. This is not being done as part of the main
OpenBSD development effort. We hope to be able to get this committed when it is
ready but we are nowhere near that yet.


getting plic working so that we can use virtio disks.

The link above is from one of my students that is working on this. This is not
in the main tree, and I'm not sure what it will take to get it there (we are
using a newer version of clang than is in base).

-ml



That's great to hear, I'm excited to see what comes of this work, hats 
off to you and your students!


It's a shame that clang was re-licensed, I hope we don't end up with a 
repeat of the gcc 4.x saga. I've heard a number of people lamenting 
about various improvements made in clang post relicencing, especially 
for non x86 arches such as powerpc.


Jordan





Re: experience setting up a low memory machine

2020-03-13 Thread Jordan Geoghegan




On 2020-03-13 18:31, Stuart Henderson wrote:

On 2020-03-13, Jordan Geoghegan  wrote:

I wouldn't get too excited about running on low memory machines. The
more RAM you can throw at something, the better, as this allows more
cache room as well as improving function of ASLR and other memory
randomizations.

It does allow more cache, but what matters for ASLR is address space, not RAM.



Ah okay, you're right. I was remembering reading something about low 
memory conditions weakening ASLR, but after doing a quick check, it 
seems it was an implementation specific issue with Windows and Linux.




Re: experience setting up a low memory machine

2020-03-13 Thread Jordan Geoghegan




On 2020-03-11 19:20, Aaron Mason wrote:

On Wed, Mar 11, 2020 at 6:47 PM Jordan Geoghegan  wrote:



On 2020-03-11 00:13, Stuart Longland wrote:

On 15/2/20 6:43 pm, Dumitru Moldovan wrote:

[SNIP]

[SNIP]

Sometimes it's better to realise when something has past its prime.

A year or two ago I had OpenBSD working on my iBook with 64MB of RAM,
even got FVWM working on it. For fun and testing purposes, I ran some
small OpenBSD virtual machines with 64MB RAM as well. A few years back I
got OpenBSD to boot with 32MB, but it wasn't particularly usable. I've
found 128MB to be usable for basic terminal work, but you're definitely
correct about 256MB being the bare minimum for anything fancy or GUI
related.



At work I run OpenBSD 6.1 in a VM for Request Tracker.  It has 512MB
RAM and it seems that may very well be overkill.  At previous jobs I
can ManageEngine ServiceDesk Plus and even in Linux you needed 2GB
minimum just for it to get out of bed.  I plan on rebuilding it with
6.6 (can't update RT because packages are too old in 6.1) and might
run it on 256MB for shits and giggles.



I wouldn't get too excited about running on low memory machines. The 
more RAM you can throw at something, the better, as this allows more 
cache room as well as improving function of ASLR and other memory 
randomizations.




Re: riscv

2020-03-13 Thread Jordan Geoghegan




On 2020-03-13 09:50, Christian Weisgerber wrote:

On 2020-03-13, "Peter J. Philipp"  wrote:


Any developer working on a riscv port and willing to share their unofficial
work for possible future collaboration?

I think I'd have heard by now if somebody was, so I'll go out on a
limb and say no, nobody's working on a RISC-V port.



I stumbled across this a while back, this guy at least claims to be 
attempting a port to RISC-V...


https://github.com/MengshiLi/openbsd-riscv-notes



Re: pf-badhost-0.3 released

2020-03-11 Thread Jordan Geoghegan




On 2020-03-11 12:41, Anders Andersson wrote:

On Tue, Mar 10, 2020 at 10:53 PM Jordan Geoghegan  wrote:

pf-badhost and unbound-adblock are both now at version 0.3, released
earlier today.

Links to the scripts can be found here:

www.geoghegan.ca/pfbadhost.html
www.geoghegan.ca/unbound-adblock.html

Thanks, this looks very interesting! But maybe you can help answering
a question that popped up when I read your page about pf-badhost.

You mention that "Subnet aggregation is used to take the address list
and "aggregate" the addresses into the smallest possible
representation using CIDR blocks.", but I was under the assumption
that pf already did this for its tables to speed up lookups.

Is there anything preventing the aggregation code to run on every pf
table modification? Assuming an already sorted list, it shouldn't take
long to merge a new entry. Perhaps I've missed some use of pf tables
that makes this impossible or not applicable in the general case.



Hi Anders,

I am by no means an expert on the nuts and bolts of pf, but I do know 
that pf stores table data in a radix tree / radix table. By their 
nature, radix trees ignore exact duplicates, but I'm not exactly sure 
how they handle the partial overlapping of ranges. This article gives an 
easy to follow cursory overview of raddix trees if you're interested:

https://blog.sqreen.com/demystifying-radix-trees/

As far as I understand, pf makes no modifications to the contents of 
your tables, all it does is parse the list to confirm the addresses 
and/or CIDR blocks are valid. When it's looking for matches within 
ranges, it will look for the most specific match available. For example, 
if you have a list containing an overlap:

...
192.168.0.0/16
192.168.1.0/22
...
When a packet from 192.168.1.5 arrives and is processed by a rule 
referencing this table, it will match with 192.168.1.0/22. Even though 
both entries are valid and match the packet, the /22 is more specific, 
and thus the one which matches closest.


pf may do some magic optimizations under the hood that I'm unaware of, 
but at the end of the day, it does not modify the actual contents of 
your table.


The use I've found in the subnet aggregation function has been mostly 
for the purpose of keeping the list clean and tidy. I have a few 
installations where I have all the lists enabled, including the use of 
the GeoIP country blacklisting function. On these installations, subnet 
aggregation can reduce the /etc/pf-badhost.txt file from ~60,000 lines 
down to ~40,000 lines. For example, when blocking China's netblocks 
(which pulls an aggregated list of all addresses assigned to China by 
APNIC, and thus uses massive CIDR blocks of /10's etc), if any addresses 
from any of the other blocklists come from China, they will be removed 
from the list as they are already covered by the CIDR block info from 
APNIC. I run pf-badhost on a bunch of Edgerouter Lites, and I've found 
them to run better when the lists are tidy.


With regards to pf performing aggregation on all tables automatically, 
it wouldn't make sense to run the full subnet aggregation calculations 
for every table load or insertion/removal, as it can be quite CPU 
intensive. It takes less than a second to load the table on a $5 Vultr 
VPS, it takes 20-70 seconds to run the subnet aggregation (depending on 
which lists are enabled). On my Edgerouter Pro with all the lists 
enabled, it takes ~6 minutes. On my Edgerouter Lite it takes ~15 minutes 
to run (over 2 hours when using the built in Perl-based aggregator). I 
just run the aggregation function with nice and let it do its thing, its 
being called by cron in the wee hours, so I'm fine just letting it chug 
along.


Regards,

Jordan



Re: experience setting up a low memory machine

2020-03-11 Thread Jordan Geoghegan




On 2020-03-11 00:13, Stuart Longland wrote:

On 15/2/20 6:43 pm, Dumitru Moldovan wrote:

Not really, about 21 years ago I was learning to get XFree86 working,
to break free from the console on a desktop with 24MB of RAM.

It's all relative… I can recall years ago experimenting with operating
systems on old machines (even by that day's standards)… trying to get
Slackware Linux running in 4MB RAM.

Today, I'm trying to cram stuff into 32kB of RAM and thinking how 4MB
and a MMU would be luxury!  (Even RetroBSD needs 128kB.)

I'm sure someone on here will tell me how they used to flip switches on
a front panel to fill the whopping 128 bytes of RAM on their 8080 with
machine code whilst dreaming of a punched tape loader and a few more
DRAM chips.

I would suggest any "modern" multi-tasking general purpose OS such as
OpenBSD, Linux, etc, you're looking at a minimum of 256MB RAM.  In the
future, probably consider doubling that… and again.

I have an old PII 300MHz laptop with 160MB RAM, and while it boots Linux
just fine (I had it running Gentoo doing AX.25 packet radio stuff),
firing up a web browser (Firefox) to check the weather is unusably slow.

I could throw OpenBSD on it, but not sure that would really make much
difference: kernel re-linking is going to be painful on that machine.

Sometimes it's better to realise when something has past its prime.


A year or two ago I had OpenBSD working on my iBook with 64MB of RAM, 
even got FVWM working on it. For fun and testing purposes, I ran some 
small OpenBSD virtual machines with 64MB RAM as well. A few years back I 
got OpenBSD to boot with 32MB, but it wasn't particularly usable. I've 
found 128MB to be usable for basic terminal work, but you're definitely 
correct about 256MB being the bare minimum for anything fancy or GUI 
related.





pf-badhost-0.3 released

2020-03-10 Thread Jordan Geoghegan

Hey folks,

Last time I posted about this, I got a fair bit of interest and I've had 
quite a few downloads and enquiries about pf-badhost, so I figured I'd 
share here that I've updated the script.


pf-badhost and unbound-adblock are both now at version 0.3, released 
earlier today.


I highly encourage anybody running an older version of these scripts to 
update to the latest version, as I have made a number of significant 
improvements to the security and robustness of the script.


Links to the scripts can be found here:

www.geoghegan.ca/pfbadhost.html
www.geoghegan.ca/unbound-adblock.html

Regards,
Jordan




Re: Time jumping forward issue under OpenBSD 6.6 VMM

2020-03-03 Thread Jordan Geoghegan




On 2020-03-03 07:43, mabi wrote:

Hello,

I am running an OpenBSD 6.6 VMM server with a few virtual machines also running 
6.6 and noticed that one of them which is running Dovecot 2.3.9 package outputs 
the following warning every few 10-15 seconds:

dovecot: imap: Warning: Time jumped forwards 28.294585 seconds

It looks like there is a time issue on that VM although I am running the 
default ntpd of OpenBSD 6.6 and I have added the following parameter into my 
/etc/sysctl.conf on that VM:

kern.timecounter.hardware=tsc

Is there anything else I can do to avoid this time issue in my VM?

Best regards,
Mabi



The clocks are basically broken on vmm. The pvclock stuff is definitely 
an improvement, but it's still not there. There's still a ways to go 
before we have proper, reasonably accurate clocks for vmm guests.




Re: suggestions for USB printer (maybe even with scanner)?

2020-02-19 Thread Jordan Geoghegan




On 2020-02-18 21:28, Stuart Longland wrote:

On 6/2/20 7:26 am, Adam Thompson wrote:

I don't know what you need in a printer, and I don't know what you mean
by cheap, so... YMMV.

However, I've found Brother **LASER** printers to be very good, and most
of them support PCL6 and/or PS3.
For example, the HL-L2370DW can only connect via USB, and supports PCL6,
and currently sells for ~C$150-160.

Somewhat related, does this hold true for their Ethernet-connected
printers as well?

We have an Epson WF-7510 which is due for replacement and while I can
set up OpenBSD on a small box to act as a print server, having something
I can point OpenBSD's lpd or cups at over a network is helpful too.


Yes, I have a Brother laser printer that I print to using lpd. I can 
send it PDF or postscript documents (or raw text) via lpr and it prints 
just fine, without any drivers or cups or other such nonsense.




Re: rspamd stop rc script doesn't work in OpenBSD 6.6

2020-02-09 Thread Jordan Geoghegan




On 2020-02-09 11:46, aisha wrote:
You need to use pkill -9 to kill rspamd, which i think should be added 
to the stop part of the rspamd daemon.


At least this is what I have been using, any other methods would be 
nice to know.




You dont need to restart rspamd if you're just modifying a config file.

You can just run "rcctl reload rspamd" to reload your config files if 
you've made any changes.


The rspamd maintainer has acknowledged the issue, and the restarting 
issue is fixed in current.




Re: Low throughput with 1 GigE interface

2020-01-30 Thread Jordan Geoghegan




On 2020-01-30 14:29, Christian Weisgerber wrote:

On 2020-01-30, Jordan Geoghegan  wrote:


All you're doing is benchmarking the speed of iperf on that machine.

I vaguely remember a thread somewhere that concluded that one of
these network benchmark tools degenerated into a benchmark of
gettimeofday(2), which apparently is very cheap on Linux and not
cheap on OpenBSD.  So you end up measuring the performance of this
system call.

I don't remember whether it was iperf...



That sounds about right. I vaguely remember reading a thread about iperf 
on misc some time in the past year mentioning that.


While OpenBSD obviously doesn't have the same network performance as 
Linux or FreeBSD, as work continues on unlocking more of the kernel, 
things will continue to get better. I think bluhm@ regularly runs some 
automated benchmarks that show that OpenBSD maxes out at around 4-5 Gbit 
/ second throughput.




Re: Low throughput with 1 GigE interface

2020-01-30 Thread Jordan Geoghegan




On 2020-01-30 13:14, Jordan Geoghegan wrote:

On 2020-01-30 10:06, livio wrote:

@KatolaZ and @remi

Thank you for your inputs on iperf2 vs. iperf3.

After all the tests I needed a clean setup again and reinstalled both
OpenBSD and Window 10.

With the new notebook (Dell vs Lenovo) I have different results.
Dell: ~ 200Mbit/s
Lenovo: ~ 145Mbit/s

iperf2 vs. iperf3 (I also ran the corresponding version on Windows):


[snip]

All you're doing is benchmarking the speed of iperf on that machine. 
Try running the tests with tcpdump.


If this APU is meant to be a router, then you should be testing its 
forwarding performance.



Woops, I meant to say tcpbench, not tcpdump.



Re: Low throughput with 1 GigE interface

2020-01-30 Thread Jordan Geoghegan

On 2020-01-30 10:06, livio wrote:

@KatolaZ and @remi

Thank you for your inputs on iperf2 vs. iperf3.

After all the tests I needed a clean setup again and reinstalled both
OpenBSD and Window 10.

With the new notebook (Dell vs Lenovo) I have different results.
Dell: ~ 200Mbit/s
Lenovo: ~ 145Mbit/s

iperf2 vs. iperf3 (I also ran the corresponding version on Windows):


[snip]

All you're doing is benchmarking the speed of iperf on that machine. Try 
running the tests with tcpdump.


If this APU is meant to be a router, then you should be testing its 
forwarding performance.




Re: dig -p 5353 foo.bar core dumped

2020-01-21 Thread Jordan Geoghegan




On 2020-01-21 11:32, Paul de Weerd wrote:

On Tue, Jan 21, 2020 at 11:25:33AM -0800, Jordan Geoghegan wrote:
| dig doesnt core dump for me, it just prints this warning: ";; Error,
| only port 53 supported". I wonder why the error isnt being printed
| for OP.

Interesting.  Are you on -current?  I updated to the latest snapshot
yesterday and get this:

[weerd@pom] $ dig -p 1234 foo.bar @127.0.0.1
Abort trap
[weerd@pom] $ dmesg | tail -n1
dig[69548]: pledge "dns", syscall 28

You may be doing something similar yet different?

Cheers,

Paul 'WEiRD' de Weerd



I ran that command on a machine running 6.6 stable, so thats probably why.

probook$ dig -p 1234 foo.bar @127.0.0.1
;; Error, only port 53 supported



Re: dig -p 5353 foo.bar core dumped

2020-01-21 Thread Jordan Geoghegan




On 2020-01-21 11:05, Paul de Weerd wrote:

On Tue, Jan 21, 2020 at 06:58:02PM +0100, Dieter Rauschenberger wrote:
| Hi misc,
|
| on my intranet i have unbound and nsd running, both on the same
| machine. unbount is listening on port 53, nsd is listening on port
| 5353 on 127.0.0.1. If i run
|
| dig @127.0.0.1 -p 53 foo.bar
|
| everything is fine. But if I want to query nsd direct with
|
| dig @127.0.0.1 -p 5353 foo.bar
|
| dig core dumps and I can read the following line in /var/log/messages:

That's pledge, as the below kernel message explains.

| Jan 21 18:42:44 ws /bsd: dig[59239]: pledge "dns", syscall 28

Dig is only allowed to do port 53.  This can be worked around by using
dig from the bind port, or with a patch to dig that allows you to use
other ports when -p is given on the command line.

However, I'd recommend running nsd on port 53 on 127.0.0.1; there it
doesn't interfere with unbound (assuming that's not listening on
127.0.0.1).  Alternatively, you can easily add a second IP address on
your loopback interface (::2) and have nsd listen there.

| The issue can be easily reporduced with any other port. No need to
| have a nsd running.

That's because it has nothing to do with nsd but with dig.  Dig has
promised not to use ports other than 53, and then when you do use
ports other than 53, the kernel terminates the process.

Cheers,

Paul 'WEiRD' de Weerd



dig doesnt core dump for me, it just prints this warning: ";; Error, 
only port 53 supported". I wonder why the error isnt being printed for OP.




Re: Can't locate OpenBSD/Quirks.pm in @INC

2020-01-17 Thread Jordan Geoghegan




On 2020-01-17 18:10, myml...@gmx.com wrote:

HI,


I downloaded the install66.fs snapshot today, 20200117, and did a fresh
install.  Even though I got the full install set, i used http from
ftp.openbsd.org as the install source.

Installation went fine but when I tried to install packages I get the
above error.

"# pkg_add -vn pftop
quirks-3.216 signed on 2020-01-17T19:15:00Z
quirks-3.216: ok
Can't load quirk: Can't locate OpenBSD/Quirks.pm in @INC (you may need
to install the OpenBSD::Quirks module) (@INC contains:
/usr/local/libdata/perl5/site_perl/amd64-openbsd
/usr/local/libdata/perl5/site_perl /usr/libdata/perl5/amd64-openbsd
/usr/libdata/perl5) at /usr/libdata/perl5/OpenBSD/AddDelete.pm line 350.

pftop-0.7p19: ok
Merging manpages in /usr/local/man: /usr/local/man/man8/pftop.8
Extracted 252817 from 253475"


[snip]

I believe quirks gets automatically installed when you install your 
first package.




Re: OpenBSD's extremely poor network/disk performance?

2020-01-09 Thread Jordan Geoghegan




On 2020-01-09 06:22, Hamd wrote:

Joe, are you a joke? Please stop insulting me, this is not
my/your_personal_fancy_forum.

This will be my last post here in misc.

Default setups, no config. changes.
Just patches installed.
Same hardware.

FreeBSD:
freebsd@test:~ # time sh -c "dd if=/dev/zero of=test.tmp bs=4k count=5
&& sync"
5+0 records in
5+0 records out
20480 bytes transferred in 0.239590 secs (854792500 bytes/sec)
0.000u 0.195s 0:00.25 76.0% 22+198k 0+1568io 0pf+0w

Result: *854.79 MB/s disk speed*

freebsd@test:~ # uname -a
FreeBSD test.local 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 GENERIC  amd64

OpenBSD:
test$ time sh -c "dd if=/dev/zero of=test.tmp bs=4k count=5 && sync"
5+0 records in
5+0 records out
20480 bytes transferred in 12.303 secs (16645247 bytes/sec)
 0m12.32s real 0m00.13s user 0m01.28s system

Result: *16.64 MB/s disk speed*



[snip]

probook$ dd if=/dev/zero of=test.tmp bs=4k count=5 && sync
5+0 records in
5+0 records out
20480 bytes transferred in 0.731 secs (280047607 bytes/sec)

I'm getting well over 250MB/s on my laptop.

Is your hard drive attached as an sd or wd device?




Re: Odd /tmp behavior

2020-01-07 Thread Jordan Geoghegan




On 2020-01-07 11:06, Karel Gardas wrote:



On 1/7/20 7:38 PM, Jordan Geoghegan wrote:

 > Using softdep on /tmp is a silly idea. >
Why? To naive eyes it may look like a natural solution: e.g. before 
temp file is even created (on drive), it may be deleted which means 
there is no meta-data change hence speedup of operation on /tmp. In 
case of classical ffs, you will need to create file (sync meta-data 
update), save some data (async), delete file (sync meta-data update). 
But honestly still need to read the code...


softdep is very complex, and it is by no means perfect or bug free. A 
good rule of thumb is to only use softdep on larger partitions that will 
see a large number of writes (particularly smaller and/or random writes).


Softdep can chew up a fair amount of kernel memory as well, and in 
certain cases changes can take over a minute to actually make their way 
on to disk. If softdep was the magic bullet that some people think it 
is, it would be enabled by default.




Re: LibreSSL performance issue

2020-01-07 Thread Jordan Geoghegan
Is there a specific reason you're running i386 instead of amd64? And why 
are you testing this on FreeBSD? Wrong mailing list


On 2020-01-07 08:26, Joe Greco wrote:

On Tue, Jan 07, 2020 at 09:33:46AM -0600, Edgar Pettijohn wrote:

In reality, when you dig down, often you find that there's another
reason for the issue.?? I was recently trying to substitute libressl
into an openssl environment.?? Performance tanked.?? Some checking
showed the speed of "speed -evp aes-256-gcm" was way off.?? It looked
to me like it was an issue with not using AES-NI.?? I'm not going to
blame libressl for that, I just lacked the time to do a deep dive on
it to figure out what was (hopefully!) configured wrong.?? Probably
something with ia32cap or whatever the libressl equivalent is.

... JG

I believe it has something to do with actually zeroing out memory
before freeing it. Which seems like a good thing to do for crypto
stuff.

My apologies.  I posted an insufficient description of the issue as it
was intended as an argument refuting the OP.  If we want to discuss my
issue, that's fine and I welcome the input.  I normally manage to
resolve these things eventually but this stumped me a bit.

This appears to be an i386-specific issue and it is perhaps a 5:1
performance difference.

Compiled on a FreeBSD 12.1R-amd64 VM, I see exactly what I would hope
to see:

--Begin-FreeBSD-12.1R-amd64
# uname -r; uname -m
12.1-RELEASE
amd64

# libressl-3.0.2/apps/openssl/openssl speed -evp aes-256-gcm
WARNING: can't open config file: /usr/local/etc/ssl/openssl.cnf
Doing aes-256-gcm for 3s on 16 size blocks: 42776805 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 64 size blocks: 28274190 aes-256-gcm's in 3.01s
Doing aes-256-gcm for 3s on 256 size blocks: 9382555 aes-256-gcm's in 3.01s
Doing aes-256-gcm for 3s on 1024 size blocks: 2636912 aes-256-gcm's in 3.01s
Doing aes-256-gcm for 3s on 8192 size blocks: 334132 aes-256-gcm's in 3.01s
LibreSSL 3.0.2
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) 
blowfish(idx)
compiler: information not available
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192 bytes
aes-256-gcm 228204.73k   601456.74k   798353.68k   897432.60k   909765.10k

# openssl speed -evp aes-256-gcm
Doing aes-256-gcm for 3s on 16 size blocks: 40297566 aes-256-gcm's in 3.01s
Doing aes-256-gcm for 3s on 64 size blocks: 27287454 aes-256-gcm's in 3.01s
Doing aes-256-gcm for 3s on 256 size blocks: 10106391 aes-256-gcm's in 3.01s
Doing aes-256-gcm for 3s on 1024 size blocks: 2858781 aes-256-gcm's in 3.01s
Doing aes-256-gcm for 3s on 8192 size blocks: 368695 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 16384 size blocks: 184909 aes-256-gcm's in 3.01s
OpenSSL 1.1.1d-freebsd  10 Sep 2019
built on: reproducible build, date unspecified
options:bn(64,64) rc4(16x,int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192 bytes  
16384 bytes
aes-256-gcm 214362.12k   580620.32k   860172.00k   973262.71k  1006783.15k  
1007226.70k
--End-FreeBSD-12.1R-amd64

Okay, so that looks fantastic to me.  Now running it on i386 on
a VM "right next door" on the same hypervisor.

--Begin-FreeBSD-12.1R-i386
# uname -r; uname -m
12.1-RELEASE
i386

# libressl-3.0.2/apps/openssl/openssl speed -evp aes-256-gcm
WARNING: can't open config file: /usr/local/etc/ssl/openssl.cnf
Doing aes-256-gcm for 3s on 16 size blocks: 8904897 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 64 size blocks: 2387064 aes-256-gcm's in 3.01s
Doing aes-256-gcm for 3s on 256 size blocks: 603284 aes-256-gcm's in 3.01s
Doing aes-256-gcm for 3s on 1024 size blocks: 153381 aes-256-gcm's in 3.01s
Doing aes-256-gcm for 3s on 8192 size blocks: 19041 aes-256-gcm's in 3.01s
LibreSSL 3.0.2
built on: date not available
options:bn(64,32) rc4(ptr,int) des(idx,cisc,16,long) aes(partial) idea(int) 
blowfish(idx)
compiler: information not available
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192 bytes
aes-256-gcm  47427.78k50805.69k51347.19k52207.47k51858.50k

# openssl speed -evp aes-256-gcm
Doing aes-256-gcm for 3s on 16 size blocks: 32056370 aes-256-gcm's in 3.01s
Doing aes-256-gcm for 3s on 64 size blocks: 21569563 aes-256-gcm's in 3.01s
Doing aes-256-gcm for 3s on 256 size blocks: 8523369 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 1024 size blocks: 2528081 aes-256-gcm's in 3.01s
Doing aes-256-gcm for 3s on 8192 size blocks: 334502 aes-256-gcm's in 3.01s
Doing aes-256-gcm for 3s on 16384 size blocks: 167762 aes-256-gcm's in 3.02s
OpenSSL 1.1.1d-freebsd  10 Sep 2019
built on: 

Re: Odd /tmp behavior

2020-01-07 Thread Jordan Geoghegan




On 2020-01-07 09:16, Raymond, David wrote:

On an AMD-64 workstation /tmp fills up to 105% according to df,
apparently as a result of UNIX pipes in a shell script passing a whole
lot of moderately big files. Examination of /tmp with du and ls -gal
on /tmp shows no big files and trying to delete everything that is
there has no effect.  Rebooting cleans out /tmp.

I had /tmp mounted with the standard options + softdep.  I eliminated
softdep and the problem appears to have gone away.

Any ideas on what is going on with softdep here?  Dmesg shows a long
series of "/tmp file system full" messages.

Dave Raymond



Using softdep on /tmp is a silly idea. I see way too many people who 
don't understand how softdep works.




Re: Traffic prioritization inside VPN

2020-01-02 Thread Jordan Geoghegan




On 2020-01-02 11:13, Radek wrote:

what about working directly on rsync side, specifying the maximum
transfer rate? (--bwlimit option)

Setting the hard transfer rate/limit on the rsync side is not what I need. I 
want my boxes to be able to use whole available bandwidth anytime. I mean if 
other services need some bandwitdh they just get it with higher priority and my 
boxes always can use *the rest*. If there is a quiet it the network my boxes 
can use the whole highway.

On Thu, 2 Jan 2020 17:57:19 +0100
fRANz  wrote:


On Thu, Jan 2, 2020 at 3:51 PM radek  wrote:


I tried to do it by "catching" this traffic on [fw_rac]/[fw_krz] by specific 
rules [1] and setting the lowest priority fot it.
Unfortunately it doesn't seem to work as expected. Bandwidth seems to be shared 
roughly equally with other traffic (tested with pushing data (netcat) through 
VPN in the same time).
I would appreciate your advice or any clues on what I have done wrong. Thank 
you.

what about working directly on rsync side, specifying the maximum
transfer rate? (--bwlimit option)
-f





FQ-CoDel may be useful for ensuring fair sharing of bandwidth. I use it 
on some bandwidth constrained links and it works quite nicely at 
ensuring that bandwidth is fairly distributed. I've found that adjusting 
the quantum level to around 300 is nice for ensuring interactive 
applications remain responsive/low latency, by prioritizing smaller 
packets over larger bulk transfer packets.




Re: OpenBSD pf - redirect all DNS queries to local DNS server

2019-12-17 Thread Jordan Geoghegan




On 2019-12-17 13:24, jin wrote:

Sorry for top posting.

Looks like you need an ip address lists which is updated dynamically. But
this method is not like what you described here. It doesn't response back
an IP address but it does block requests which is trying to get those
ad.servers. if you wish you can choose action to "reject" instead of
"drop". I choose to reject requests because of we just have two devices at
home network.

There are couple of sites which shares ad server names and ip addresses as
a list and they update those lists. As described in below link, you can use
a script to stop traffic which you don't want to have. Basically the script
updates 'source of bad ad server list' periodically and feed your pf rules.

https://www.geoghegan.ca/pfbadhost.html

On Tue, Dec 17, 2019, 23:57 lu hu  wrote:


Our little home network:

ISP -> ROUTER -> SWITCH -> WIFI APs -> CLIENTS

ROUTER: OpenBSD 6.5, giving DHCP+fwing internet to the WIFI APs. Based on
https://www.openbsd.org/faq/pf/example1.html#pf and
https://www.openbsd.org/faq/pf/example1.html#dhcp

CLIENTS: laptops, smartphones.

So everything is going through the ROUTER.

We can see a https://www.openbsd.org/faq/pf/example1.html#dns DOC for how
to setup a DNS server, ~ok.

AD filtering. We would like to have one, but not a fancy one, just a
working one.

Based on "bad hosts", ex.: if a client queries iamAD.foo, then answer it
back as 127.0.0.1, so the clients will try to connect to themselfes, which
will end up not showing the AD.

The big question: Is there any DOC for OpenBSD about this? What pf rules
needed to redirect any DNS server (ex.: 8.8.8.8 or 1.1.1.1) requests to the
DNS server running on the ROUTER, coming from the CLIENTS?

So ex.: if a smartphone CLIENT wants to query iamAD.foo domain to get ADs,
it will only get back 127.0.0.1





Hey, I'm the author of that script. If you're looking to block ads via 
DNS, geoghegan.ca/unbound-adblock.html may be what you're looking for.


It pulls a popular ad server blocklist and makes unbound return NXDOMAIN 
when a device tries to query a known ad server. Certain devices have 
issues when redirecting their querys to 127.0.0.1 or 0.0.0.0, and some 
devices may waste time retrying queries for a long period of time. 
Setting static redirects to a particular address causes unbound to eat 
up a ton of memory, wheras returning NXDomain uses almost no memory.


Cheers,

Jordan



Re: unbound network optimizations

2019-12-12 Thread Jordan Geoghegan




On 2019-12-12 06:21, Winter Paulson wrote:

Hello,

I'm also experiencing the "Host is down" problem:

unbound: [85343:0] error: recvfrom 361 failed: Host is down

Running openbsd 6.6 (GENERIC.MP), current syspatch,
native unbound as a full resolver, pf disabled.

OpenBSD is a guest VM on a debian buster host using virtual e1000
network card ("Intel 82540EM" driver in openbsd). No firewall
in between. The VM is a tor-exit node.


I've heard others recommend using the vio driver over the em driver 
numerous times on here if running a virtualized instance. You may have a 
better time than you are now by using the VirtIO drivers. The intel nic 
emulation can sometimes have issues. Better to use an interface designed 
for virtualized environments.



Cheers,

Jordan



Re: Can't select files to upload in a browsers

2019-12-06 Thread Jordan Geoghegan




On 2019-12-06 02:56, dmitry.sensei wrote:

Firefox and Chromium browser, in the file selection window for upload,
does not show the contents of directories other than the Downloads
directory


OpenBSD 6.6-current
OpenBSD 6.6 GENERIC.MP#509 amd64

openbox-3.6.1p7 small, fast & usable window manager
firefox-71.0Mozilla web browser
chromium-78.0.3904.106 Chromium browser
gtk+2-2.24.32p8 multi-platform graphical toolkit
gtk+3-3.24.13   multi-platform graphical toolkit
gtk+4-3.96.0p6  multi-platform graphical toolkit



That's because both browsers are using unveil.



Re: sysupgrade to 6.6 failed at comp66.tgz

2019-11-23 Thread Jordan Geoghegan



On 2019-11-23 14:20, Rachel Roch wrote:



This topic has been beat to death. deraadt@ and other have made it clear that 
if you do not install all the sets, you are running an unsupported 
configuration. It has been stated that if people keep bitching, they're just 
going to merge the release sets into one set.

I like the fact that there are separate sets. A number of times I've had to 
squeeze an install onto a <2GB disk, and it was useful being able to select 
only the specific sets I wanted/needed, while at the same time acknowledging that 
it was indeed an unsupported configuration.

If people are going to try and be edgelords by refusing to install all the 
sets, then it's up to them to maintain and diagnose their unsupported 
configuration.


You can't seriously be calling "-x* -game*" an unsupported configuration ?  
Seems to me like a sensible thing to do on any box that's going to be headless for its 
entire life and only ever accessed via SSH (or text console at a push).



Running without the X sets is most certainly an unsupported 
configuration. Many packages/ports have dependencies on things in the X 
sets, I've been bit by this issue my self. Certain ports have "no_x11" 
flavours, but it's not a guarantee.


With regards to the game sets, they're less than 5MB, so its pretty 
irrelevant.


Forgoing the X sets does nothing for security, and ultimately further 
removes you from a standard supported install. Unless you're trying to 
do an install on a super small disk, just install all the sets. If 
you're running any sort of modern or important production machine, 
you're going to have more than enough disk space to install all the sets.


If you want to be an edgelord and not install all the sets, then by all 
means, please enjoy your unsupported system. Don't come back bitching 
and moaning if something breaks.




Re: sysupgrade to 6.6 failed at comp66.tgz

2019-11-23 Thread Jordan Geoghegan



On 2019-11-23 13:45, Rachel Roch wrote:



- maybe sysupgrade needs to be patched to avoid this issue?


Probably not. sysupgrade has assumptions baked in to it which have
evidently been rendered invalid either by another tool or by the
person using them. That tool is where the patch most likely ought
to be directed.




If I may make a little comment here.

Surely it is a little bit questionable to "bake assumptions" into sysupgrade 
that everybody is going to do a complete install when the OpenBSD installer itself gives 
you the option to select what is going to be installed.

At the very least, may I suggest that even if the developers don't want to increase 
the intelligence of sysupgrade that they at least code in some sanity checks (e.g. 
"pick a file - or two - at random from the core tgz files that you would 
normally expect to be present on the system if a 'full-default' install was done.  
If file not present, then throw a horrid error message and abort).

It strikes me as a little silly to put a tool out there that you know will trash (or at 
least severely brick) a user's system just because of some severely opinionated 
"baked assumptions" coded into it.



This topic has been beat to death. deraadt@ and other have made it clear 
that if you do not install all the sets, you are running an unsupported 
configuration. It has been stated that if people keep bitching, they're 
just going to merge the release sets into one set.


I like the fact that there are separate sets. A number of times I've had 
to squeeze an install onto a <2GB disk, and it was useful being able to 
select only the specific sets I wanted/needed, while at the same time 
acknowledging that it was indeed an unsupported configuration.


If people are going to try and be edgelords by refusing to install all 
the sets, then it's up to them to maintain and diagnose their 
unsupported configuration.




Re: Running Windows inside vmm/vmd VM.

2019-11-22 Thread Jordan Geoghegan



On 2019-11-22 19:53, Jes wrote:

On Fri Nov 22, 2019 at 9:04 PM Dumitru Moldovan wrote:

  Supported guest operating systems are currently limited to OpenBSD and
  Linux. As there is no VGA support yet, the guest OS must support serial
  console.

Speaking of this, does anyone here have any experience running Linux VMs
on vmm/vmd? I threw Alpine/Debian installs together recently and
they seemed to work well. Looking for anyone with longer-term experience
as I'm interested in setting up a VPS hosting service on vmm/vmd, and
would appreciate any advice or anecdotes.

I have some Alpine and Void Linux installs running on vmm. They work 
well, with some caveats.


You may have issues with your VMs clocks. OpenBSD guests in vmm are now 
able to use the pvclock driver, which has greatly improved time keeping 
on my VMs, although I still do have some erratic clock jumping, but at 
least it's not so bad that ntpd can't keep up with it.


However the timekeeping situation for my Linux VMs is bleak. On both 
Void and Alpine, no clocks are even detected. In the dmesg it complains 
about the TSC clock source being unstable. Ultimately, we're left with 
only jiffies as a clock source option:


void$ cat 
/sys/devices/system/clocksource/clocksource0/available_clocksource

refined-jiffies jiffies

As a result, my clocks run at about one third of real time.

I've tried the Linux VM's on both an old Xeon machine as well as a 
modern Ryzen machine, and the clock situation seems to be equally bad on 
both of them.


...

Clock issues aside, I've found Linux guests to get better networking 
throughput on vmm than OpenBSD guests.


A few results from benchmarking Alpine vs Void vs OBSD, with iperf3:

(vmm host is older xeon rig, iperf3 tester is ryzen desktop)

Alpine got this result:

[ ID] Interval   Transfer Bitrate
[  5]   0.00-10.00  sec   511 MBytes   429 Mbits/sec  sender
[  5]   0.00-8.60   sec   511 MBytes   499 Mbits/sec  
receiver


Void Linux Got this result:

[ ID] Interval   Transfer Bitrate
[  5]   0.00-10.00  sec   611 MBytes   512 Mbits/sec  sender
[  5]   0.00-7.00   sec   610 MBytes   732 Mbits/sec  
receiver


And OpenBSD got this result:

[ ID] Interval   Transfer Bitrate
[  5]   0.00-10.00  sec   299 MBytes   251 Mbits/sec  sender
[  5]   0.00-10.19  sec   299 MBytes   246 Mbits/sec  
receiver


Because folks always freak out when tcpbench is forgotten about, I 
tested tcpbench as well between the two machines running OpenBSD:


Peak Mbps:  231.240 Avg Mbps:  204.423

I know that was some very unscientific testing, but hey, you asked for 
anecdotes.


Cheers,

Jordan



Re: Home NAS

2019-11-15 Thread Jordan Geoghegan



On 2019-11-15 20:47, Predrag Punosevac wrote:

Jan Betlach wrote:

[snip]


2. A HP P222 array controller works right out of the box on
OpenBSD, maybe FreeBSD as well but the combination of ZFS and RAID
controller seems weird to me.


FreeBSD has a better support for HWRaid cards than OpenBSD. I am talking
about serious HWRaid cards like former LSI Controllers. Only Areca used
to fully support OpenBSD. Also FreeBSD UFS journaling is more advanced
than OpenBSD journaling.


OpenBSD's UFS doesn't do any journalling.

[snip]


3. OpenBSD is actually out of my expectation. CIFS and NFS is just
easy to setup. The most fabulous thing to me is the full disk
encryption. I had a disk failure and the array controller was burnt
once because I had some cooling issue. However, I was confident to get
a replacement and no data was lost.


OpenBSD NFS server implementation is slow comparing to others but for
home users YMMV.
I was able to get Gigabit line rate from an OpenBSD NAS to CentOS 
clients no problem. The OpenBSD NFS client is admittedly somewhat slow-- 
I was only able to get ~70MB/s out of it when connected to the same NAS 
that gets 100MBps+ from Linux based NFS clients.


Code:
# bioctl sd4
Volume  Status   Size Device
softraid0 0 Online  2000396018176 sd4 RAID1
   0 Online  2000396018176 0:0.0   noencl 
   1 Online  2000396018176 0:1.0   noencl 

is very crude. It took me 4 days to rebuild 1TB mirror after accidental
power off one HDD. That is just not something usable for a storage
purpose in real life.


I have an OpenBSD NAS at home with 20TB of RAID1 storage comprised of 10 
4TB drives. Last time I had to rebuild one of the arrays, it took just 
under 24 hours to rebuild. This was some months ago, but I remember 
doing the math and I was getting just under 50MB/s rebuild speed. This 
was on a fairly ancient Xeon rig using WD Red NAS drives. If it took 
your machine 4 days to rebuild a 1TB mirror, something must be wrong, 
possibly hardware related as that's less than 4MB/s rebuild speed.




At work where I have to store petabytes of data I use only ZFS. At home
that is another story.

For the record BTRFS is a vaporware and I would never store the pictures
of my kids to that crap.

Cheers,
Predrag


Cheers,

Jordan



Re: Skype alternatives for OpenBSD

2019-11-03 Thread Jordan Geoghegan



On 2019-11-03 05:15, Stefan Sperling wrote:

On Sat, Nov 02, 2019 at 02:47:16PM -0700, Jordan Geoghegan wrote:

Assuming Firefox or chromium on OpenBSD has WebRTC support (havent checked
in a while), talky.io should work. It's a free website that supports WebRTC
chats. I've used it in the past with great success.

The www/nextcloud port with the 'Talk' add-on, and with telephony/turnserver,
can be used to self-host a WebRTC server on OpenBSD.


That's really cool, thanks for mentioning that. I'm going to go try 
setting up a Nextcloud WebRTC server now :p




Re: Tools for writers

2019-11-03 Thread Jordan Geoghegan



On 2019-11-02 18:29, Ingo Schwarze wrote:

Hi Jordan,

Jordan Geoghegan wrote on Sat, Nov 02, 2019 at 05:44:23PM -0700:


I've thought about learning latex and mandoc and all the fancy
tools, but I've just never gotten around to it.

Actually, both mandoc(1) and mdoc(7) are off-topic in this thread.
You cannot use either for writing a book, neither the mdoc(7)
language nor the mandoc(1) program supports any of the important
features.

Woops, I got pandoc and mandoc confused.

That said, the obvious answer for the OP is of course the
"textproc/groff" port (disclosure: which i maintain).  The roff(7)
language and the troff programm is what people in the UNIX world
always used for writing books and journal articles, and it is very
much alive even after the roff language celebrated its 55th birthday
this year.  I'm in the habit of using it to prepare slides for
conference talks (with textproc/gpresent), for example, and i'm not
the only only one.

[snip]

As long as you only *use* macro packages, groff is *much*
easier to use than LaTeX (not least because the quality of
documentation of groff is vastly superior to LaTeX, and LaTeX
documentation is so extremely huge and fragmented that it's
a terrible challenge to find anything you need).

[snip]

Most certainly, it is *much* easier to get good typography out
of groff or LaTeX (no matter which one) than out of LibreOffice
or any similar abomination.

Yours,
   Ingo


Thanks for the recommendation Ingo, I'm going to test out groff for a 
writing project I have coming up.


Cheers,

Jordan



Re: Tools for writers

2019-11-02 Thread Jordan Geoghegan



On 2019-11-02 15:54, Marc Chantreux wrote:

hello,


You can't go wrong with LibreOffice. I've written thousands of pages over
the years with it. It may be too "heavy" for some, but for me, if I'm doing
something too complex for vi or mousepad, I just fire up LibreOffice.

to me there is no such thing that is too complex for the unix documentation
toolchain that you can achieve with libreoffice. i feel the other way
around: libreoffice is always a bad choice:

* when i need a rich good looking document in which you can
   easily add graphical material of very different nature (music cheets,
   chemical or math formula, gantt graph, ...), then

   [your editor of choice] + pandoc + git + latex + tikz + gnuplot + graphviz + 
m4

   is really the best thing i found

* if you need interactivity and animation (which isn't my case): the web is 
there
   (i personally use pandoc + pug + livescript (to be replaced by elm) +
   stylus)

the only one case where libreoffice is the good choice is if you mind
the learning curve but writting a book is a long process, pay the bill
at first to be more peaceful later seems to be a good deal to me.

regards.
marc



Fair enough, I have no issue with using the various unix tools for 
making documents, but for my use case, LibreOffice has treated me well. 
I primarily use it for simple things like putting together invoices, 
writing articles, rendering documents to PDF or postscript, and reading 
.docx files people send me. I'm sure there's a superior way to do all 
this, but I'm lazy and I like some of the more mundane features of 
LibreOffice like being able to render a document to postscript and then 
print to the printer I have specified in my /etc/printcap file as well 
as .docx format support for when I have to collaborate with average 
people. I've thought about learning latex and mandoc and all the fancy 
tools, but I've just never gotten around to it.




Re: Tools for writers

2019-11-02 Thread Jordan Geoghegan



On 2019-11-02 13:18, Chris Bennett wrote:

On Sat, Nov 02, 2019 at 03:16:22PM -0400, STeve Andre' wrote:


On 2019-11-02 15:07, Antoine Jacoutot wrote:

You obviously never wrote a book.
At least not with the requirements OP asked for. >

Actually, I am, right now.  I've found that "formatting" is an
annoyance, when writing material.  Get it written, *then* worry
about how it looks.  I've done this for more than 40 years when
creating documents, reports and such for work.

--STeve Andre'



Actually the OP said that not necessarily the same application.
I have to agree that writing the content and doing the formatting to be
two separate processes. I write in a furious stream of conciousness.
Looking at the formatting, etc. just impedes my work.
Spout out the thoughts. Then review content. Then work on formatting and
editing content.

/bin/ed

Then do the delicate work of editing and formatting after the creative
stream ends. Or creativity may be lost. Forever. Over seeing some
misspelled word or wrong punctuation.

Two or three different tools. Since there are really about three
processes being done that are quite different. For me, multitasking
sucks.

But, please, what is good for formatting? I don't have an answer for
that myself. I am considering writing as a new direction for myself.
Getting old sucks.

Chris Bennett



You can't go wrong with LibreOffice. I've written thousands of pages 
over the years with it. It may be too "heavy" for some, but for me, if 
I'm doing something too complex for vi or mousepad, I just fire up 
LibreOffice.




Re: Skype alternatives for OpenBSD

2019-11-02 Thread Jordan Geoghegan
Assuming Firefox or chromium on OpenBSD has WebRTC support (havent 
checked in a while), talky.io should work. It's a free website that 
supports WebRTC chats. I've used it in the past with great success.



On 2019-11-02 14:35, Jonathan Drews wrote:

Is there an alternative to Skype that runs on OpenBSD? I looked in 
http://openports.se/
and didn't see anything. I want to take online classes nad need a video
conferencingsoftware. --Kind regards,Jonathan




Re: rspamd stop rc script doesn't work in OpenBSD 6.6

2019-10-27 Thread Jordan Geoghegan



On 2019-10-27 17:29, Chris Narkiewicz wrote:

Rspamd stop rc script doesn't work in OpenBSD 6.6.

1. Fresh OpenBSD 6.6 installation
2. pkg_add rspamd
3. rcctl start rspamd

Works.

4. rcctl stop rspamd timeouts

Looking at rspamd logs, it looks like it doesn not work
well with SIGTERM. It waits for workers.

Currently I work around it by adding custom rc_stop():

rc_stop() {
    pkill -KILL -T "${daemon_rtable}" -xf "${pexp}"
    pkill -KILL -u _rspamd
}


Anybody can confirm if this is a problem?



Yes, can confirm. I think I read something the other day mentioning 
sthen@ possibly having a diff floating around to fix the issue. 
Hopefully the fix will be backported to -stable.




Re: Recommended web and database server specification

2019-08-14 Thread Jordan Geoghegan



On 8/14/19 9:40 PM, Tito Mari Francis Escano wrote:

Hi Jordan,

Thanks for replying. I apologize I forgot to mention that I was asking 
for the web and database spec because this will be an on-premise 
deployment, cloud platforms are out of the equation. I hope this will 
clarify my concern asking for server specification for a web and 
database server that won't have high traffic but will have a long run.


Thanks again.

On 15/08/2019 12:27 PM, Jordan Geoghegan wrote:


On 8/14/19 7:56 PM, Tito Mari Francis Escano wrote:

Hi to everyone at misc,

I'm recently working on an OpenBSD-based PHP7 web application with 
PostgreSQL-backend for a local government agency and was wondering 
what would you recommend as the acceptable server specification. 
This web application won't reach the Google or Facebook level of 
visits per day, but I was hoping to prepare this be deployed and run 
for quite a long time and ready for about 60,000 visits per day at 
most.


Your advise and recommendation would be greatly appreciated. Thanks 
so much.





I don't have much experience running php apps in production, but I 
have run a Django app in production using httpd and Postgres. Using a 
basic 4 core VM from Vultr, benchmarks showed it was able to serve 
over 4000 requests per second. With those kind of numbers you could 
easily serve over a million hits per day. It didn't have a lot of 
complex DB activity going on, so the benchmarks were more testing 
httpd and Djangos throughput, but I digress. If you cluster your 
servers, you should be able to hit far more.


Jordan



If you're only planning on serving ~60,000 hits a day, any modern server 
should suit your needs adequately. I'm sure others on misc@ are far more 
well versed in hardware than I am, but I've had good success with Dell 
and HP machines in the past. Any recent sever should do. A single socket 
8 core server and 64GB of ECC ram and some RAID-1 backed SSD storage 
should treat you well. I'm no expert when it comes to DB architecture, 
but I imaging fast storage and high frequency cores would be the name of 
the game there.


Cheers,

Jordan



Re: Recommended web and database server specification

2019-08-14 Thread Jordan Geoghegan



On 8/14/19 7:56 PM, Tito Mari Francis Escano wrote:

Hi to everyone at misc,

I'm recently working on an OpenBSD-based PHP7 web application with 
PostgreSQL-backend for a local government agency and was wondering 
what would you recommend as the acceptable server specification. This 
web application won't reach the Google or Facebook level of visits per 
day, but I was hoping to prepare this be deployed and run for quite a 
long time and ready for about 60,000 visits per day at most.


Your advise and recommendation would be greatly appreciated. Thanks so 
much.





I don't have much experience running php apps in production, but I have 
run a Django app in production using httpd and Postgres. Using a basic 4 
core VM from Vultr, benchmarks showed it was able to serve over 4000 
requests per second. With those kind of numbers you could easily serve 
over a million hits per day. It didn't have a lot of complex DB activity 
going on, so the benchmarks were more testing httpd and Djangos 
throughput, but I digress. If you cluster your servers, you should be 
able to hit far more.


Jordan



Re: IPv6 problems

2019-08-13 Thread Jordan Geoghegan



On 8/13/19 10:11 AM, Thomas Bohl wrote:

Hello,


My hostname.vio0 looks like this:


dhcp

inet6 alias  64



You most likely need to add a route. Add something like this to your 
hostname file:

!route add -inet6 default fe80::1%vio0


Just in case you have the same problem. For whatever reason, after a 
reboot, I have to do this in order to get IPv6 traffic flowing:

ping6 -c 10 fe80::1%vio0


or just add your gateway to your /etc/mygate file.




AMDGPU Freesync Supported?

2019-06-09 Thread Jordan Geoghegan

Hello,

I noticed that amdgpu is being ported over to OpenBSD, and I was curious 
if Freesync will be supported by the newly ported code? Any info/insight 
folks could provide would be much appreciated.


Cheers,

Jordan



Re: Modern browser for OpenBSD powerpc

2019-05-25 Thread Jordan Geoghegan



On 5/25/19 4:00 AM, Riccardo Mottola wrote:

Hi,

On 5/23/19 8:19 AM, John Gould wrote:

Can someone suggest a modern graphical browser for OpenBSD PowerPC?
I'm trying to run
several G5's and g4 mini's on 6.5 as desktop machines. The basic
install works really well but there doesn't seem to be an up to date
graphically browser.

It's thanks to all the work the devs have put into OpenBSD powerpc
that these machine are still very usable. They are hopelessly out of
date as far as the Mac OS are concerned!



you might try ArcticFox, it has a decent success on Linux PowerPC. 
Several endianness fixes were imported.


It is not "totally" modern, but still more modern than Dillo. Beware 
that you need at least 1G of RAM to be of decent use with modern 
websites, 2GB is better.


Although it is of PaleMoon heritage and thus Linux/Mac heritage, I 
fixed compilation on NetBSD, OpenBSD and lately even FreeBSD compiles 
out of the box.



Beware however, that while perfectly usable on older x86, it has no 
working JIT, so JS intensive websites will be slow on PowerPC. Also 
compilation on OpenBSD/ppc was never attempted by me, only on 
Linux/PPC. OpenBSD amd64 however does work.



Riccardo

[1] : Official Repo: https://github.com/wicknix/Arctic-Fox

[2] : My current working fork, which gets regularly pulled into main: 
https://github.com/rmottola/Arctic-Fox






If you're going down that path, you should see if you can get TenFourFox 
to compile. TenFourFox does have a jit and supports altivec.



http://www.floodgap.com/software/tenfourfox/

https://github.com/classilla/tenfourfox



Re: Modern browser for OpenBSD powerpc

2019-05-23 Thread Jordan Geoghegan



On 5/22/19 11:19 PM, John Gould wrote:

Can someone suggest a modern graphical browser for OpenBSD PowerPC?


Dillo is a pretty good browser that works on macppc. It doesn't support 
javascript, but is perfectly adequate for reading non-terrible websites.




Re: Booting octeon in single user mode

2019-05-21 Thread Jordan Geoghegan
Thanks Stuart, this also works well and saves having to mess around with 
uboot.


Cheers,

Jordan

On 5/7/19 3:58 PM, Stuart Henderson wrote:

On 2019-05-07, Jordan Geoghegan  wrote:

Hi folks,

I have an old Edgerouter Lite I set up last year that I've forgotten the
passwords for.

I know you can boot single user mode on amd64 by typing "boot -s" at the
bootloader prompt, but that does not seem to exist on octeon.

Any help you guys can provide getting my octeon into single user mode
would be much appreciated.

Jordan



Untested on Octeon, but usually you can Ctrl-\ before it goes multiuser.






Re: Booting octeon in single user mode

2019-05-21 Thread Jordan Geoghegan

Thanks Visa, that did the trick!

Sorry for the late reply, just got a chance to look at the machine, and 
everythings working great now.


Jordan

On 5/7/19 9:58 AM, Visa Hankala wrote:

On Tue, May 07, 2019 at 09:19:05AM -0700, Jordan Geoghegan wrote:

I have an old Edgerouter Lite I set up last year that I've forgotten the
passwords for.

You can reset the passwords by using bsd.rd. Run the upgrader until
it has mounted the filesystems and asks "Location of sets?"
Type ! and press Return to enter shell, and run "chroot /mnt". After
that, you can use passwd(1) as usual.





Re: Haskell compilation issues

2019-05-19 Thread Jordan Geoghegan



On 5/19/19 1:24 PM, Matthias Kilian wrote:

On Sun, May 19, 2019 at 12:18:44AM -0500, Joe Nelson wrote:

Matthias Kilian wrote:

ps: please note that I'm not subscribed to misc@ with my 'real'
mail account, only with a crappy gmail account I'm only reading on
my tablet (from which I forwarded your mail to my real address). So
better cc' me if you've any other questions ;-)

FYI, the way you replied to the original message broke the reply chain
in my mail reader (Mutt). You should include an "In-Reply-To" message
header that references the Message-ID of the mail to which you are
replying. I know it might be a little tricky from your tablet setup, but
wanted to bring it to your attention if you weren't aware.

I'm very well aware of it, and I've a really smart solution for the
problem: I'll never ever even try to give some helpful answer to
any of the mails on misc@



No need to get all triggered, the guy is just giving you a heads up.



  1   2   >