Re: copyright notice question

2020-10-21 Thread John-Mark Gurney
Rick Macklem wrote this message on Wed, Oct 21, 2020 at 01:03 +:
> Warner Losh wrote:
> >On Mon, Oct 19, 2020, 7:25 PM Rick Macklem 
> >mailto:rmack...@uoguelph.ca>> wrote:
> >>I'll admit I've hesitated to ask this for a long time, but I guess I have 
> >>to;-)
> >>I've created two daemons for NFS-over-TLS, using the code in
> >>/usr/src/usr.sbin/gssd/gssd.c as a starting point.
> >>--> As such, I left the copyright notice from this file on the two files.
> >>  (As you can see, it is a 2 clause FreeBSD one, so the terms aren't
> >>   an issue.)
> >>
> >>What I am wondering is if I should be adding my name to it as an
> >>additional author or something like that?
> >>(I don't care, but it does seem a little weird that the copyright is held
> >> by Isilon Inc, since I have no connection to them.)
> >>
> >Likely. If you changed a substantial amount, then yes. The rule of thumb is 
> >>50%
> > is no brainer yes. Smaller percentages require more nuanced judgement as to 
> > whether the changes are substantial enough to create a new work. Chances are
> > quite good you fall into one of those categories. Also, if you have 
> > replaced more 
> >than ~90% chances are good no original work remains. Again, a judgement 
> >call. 
> >There are more technical legal guidelines, but that would require a lawyer.
> >
> >My hunch is that unless this is something far more trivial than I then I'd 
> >add the
> > notice, but retaining the old.
> Well, I'd guess it's somewhere in the 50->90% category.
> Would just adding a comment below the current copyright notice like:
> /*
>  * Extensively modified from /usr/src/usr.sbin/gssd.c for RPC-over-TLS
>  * by Rick Macklem.
>  */
> be sufficient for the project, or should I put a second copyright notice
> on it with my name on it? (This will seem odd, since it will have the same
> terms as the extant one, but if that is what is appropriate for the project..)

I'd add both the above notice, but also your own copyright at the top...
You don't need to add your own copyright, but if you do, the by Rick
Macklem isn't needed in the modified from notice (and IMO, an
unnecessary distraction)...  The fact that you did the initial
modifications will be apparent in the SVN/git history.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


RFC: allow first boot config from msdos partition

2020-10-08 Thread John-Mark Gurney
Hello,

I've had the idea that it'd be nice to allow more first boot
configuration of our ARM images from the FAT partition.  The machine
I normally use is a MacOSX box, so when writing images, it isn't easy
to put an rc.conf on the image.

This change imports configinit from cperciva's ec2-scripts (one minor
tweak to make it better), and creates an fs_configinit script that
reads the config.init file from /boot/msdos and passes it through
configinit.

This will allow first boot config, like setting a static IP, installing
packages at first boot making it easier to get a system functional.
It also means that if you have a simple appliance, if you package your
config as a config.init script, it's easier to upgrade and test FreeBSD
snapshots and releases than it was previous.

I've written the man pages.  If there is anything that isn't clear or
missing, please let me know.

The changes are in here: https://reviews.freebsd.org/D26713

Thanks!

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Deprecating ftpd in the FreeBSD base system?

2020-09-17 Thread John-Mark Gurney
Rodney W. Grimes wrote this message on Thu, Sep 17, 2020 at 10:53 -0700:
> > FTP is firewall unfriendly.
> 
> Passive mode solved that decades ago.

Requires that the server not be behind a firewall or port forwarding
as well..

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Deprecating ftpd in the FreeBSD base system?

2020-09-17 Thread John-Mark Gurney
Ian Lepore wrote this message on Thu, Sep 17, 2020 at 09:01 -0600:
> On Thu, 2020-09-17 at 18:43 +0400, Gleb Popov wrote:
> > On Thu, Sep 17, 2020 at 6:05 PM Cy Schubert <
> > cy.schub...@cschubert.com>
> > wrote:
> > 
> > > I've been advocating removing FTP (and HTTP) from libfetch as well.
> > > People
> > > should be using HTTPS only.
> > > 
> > 
> > Isn't this a bit too much? I often find myself in need to download
> > something starting with "http://; or "ftp://; and use fetch for this.
> 
> Indeed, we have products which rely on this ability in libfetch and we
> have to keep supporting them for many many years to come.
> 
> I hate it when someone imperiously declares [For security reasons]
> "People should/shouldn't be using __".  You have no idea what the
> context is, and thus no ability to declare what should or shouldn't be
> used in that context.  For example, two embedded systems talking to
> each other over a point to point link within a sealed device are not
> concerned about man in the middle attacks or other modern internet
> threats.

And I really dislike when people want to make sure that their unique
case that less than a percent of people would every hit blocks the
security improvements for the majority of people...

I've given up on a number of security improvements in FreeBSD because
of this attitude...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Deprecating ftpd in the FreeBSD base system?

2020-09-17 Thread John-Mark Gurney
Warner Losh wrote this message on Thu, Sep 17, 2020 at 10:08 -0600:
> On Thu, Sep 17, 2020 at 8:05 AM Cy Schubert 
> wrote:
> 
> > I've been advocating removing FTP (and HTTP) from libfetch as well. People
> > should be using HTTPS only. (libfetch could support a plugin that might be
> > supplied by a port should someone be inclined to write one.)
> 
> The project isn't going to do that. "tools not policy" dictates that
> anything like that should be done in fetch(1) and likely only as a command
> line option for people that require a secure connection (or that can
> tolerate an insecure one).

Do we have a way for the admin/root to set fetch's policy to block FTP
and HTTP?

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: CFT: major update to if_ure

2020-09-11 Thread John-Mark Gurney
John-Mark Gurney wrote this message on Sat, Jul 25, 2020 at 16:13 -0700:
> Hello,
> 
> I'd like people who have ure (RealTek) based USB devices to test
> review D25809[0].
> 
> This update adds support for:
> - HW VLAN tagging
> - HW checksum offload for IPv4 and IPv6
> - tx and rx aggreegation (for full gige speeds)
> - multiple transactions
> 
> In my testing, I am able to get 900-950Mbps depending upon
> TCP or UDP, which is a significant improvement over the previous
> 91Mbps (~8kint/sec*1500bytes/packet*1packet/int).
> 
> Thanks.
> 
> [0] https://reviews.freebsd.org/D25809

This has now landed in:
https://reviews.freebsd.org/rS365648

Let me know if there are any regressions.

Thanks.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: rfc: should extant TLS connections be closed when a CRL is updated?

2020-09-04 Thread John-Mark Gurney
Rick Macklem wrote this message on Fri, Sep 04, 2020 at 01:20 +:
> The server side NFS over TLS daemon (rpc.tlsservd) can reload an updated
> CRL (Certificate Revocation List) when a SIGHUP is posted to it.
> However, it does not SSL_shutdown()/close() extant TCP connections using TLS.
> (Those would only be closed if the daemon is restarted.)
> 
> I am now thinking that, maybe, an SSL_shutdown()/close() should be done on
> all extant TCP connections using NFS over TLS when an updated CRL is loaded,
> since a connection might have used a revoked certificate for its handshake.
> 
> What do others think?

IMO, this should scan the existing connections, and only shut them
down if they are using a revoked Cert.  This is the correct way to
do things.

I do realize that this is likely not possible, and in reality, the
ssl library in use should do this automatically, but likely does not.
As the library likely does not, we should probably make this an
option to close all connections upon CRL reload, with it being well
documented.

Now that option should likely be set to default on, but documented
such that if you do regular/often CRL reloads, that a user may want
to turn that off if it's disruptive to their server.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: CFT: major update to if_ure

2020-08-24 Thread John-Mark Gurney
Ganbold Tsagaankhuu wrote this message on Wed, Aug 19, 2020 at 16:27 +0800:
> On Tue, Jul 28, 2020 at 2:35 AM John-Mark Gurney  wrote:
> 
> > Ganbold Tsagaankhuu wrote this message on Mon, Jul 27, 2020 at 18:29 +0800:
> > > On Mon, Jul 27, 2020 at 5:14 AM John-Mark Gurney 
> > wrote:
> > >
> > > > Ganbold Tsagaankhuu wrote this message on Sun, Jul 26, 2020 at 11:05
> > +0800:
> > > > > On Sun, Jul 26, 2020 at 7:13 AM John-Mark Gurney 
> > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I'd like people who have ure (RealTek) based USB devices to test
> > > > > > review D25809[0].
> > > > > >
> > > > > > This update adds support for:
> > > > > > - HW VLAN tagging
> > > > > > - HW checksum offload for IPv4 and IPv6
> > > > > > - tx and rx aggreegation (for full gige speeds)
> > > > > > - multiple transactions
> > > > > >
> > > > > > In my testing, I am able to get 900-950Mbps depending upon
> > > > > > TCP or UDP, which is a significant improvement over the previous
> > > > > > 91Mbps (~8kint/sec*1500bytes/packet*1packet/int).
> > > > >
> > > > > Does performance improve for if_ure device on USB2?
> > > > > I will try to test it in a couple of days on NanoPI R1 and R1S
> > boards.
> > > >
> > > > Yes, it should.
> > > >
> > > > I never tested the before driver on USB2, but I'm now able to get
> > > > 211Mbps TX and 190Mbps RX TCP, and 227Mbps TX and 225Mbps RX UDP.
> > > >
> > > > I believe it is likely that the same 91Mbps speed limit applied to
> > > > USB2 as well.
> > >
> > > Couldn't find your iperf test scripts and I tested only tcp:
> >
> > My test script isn't performance, just features, and I'm thinking about
> > how/where to publish it...
> >
> > You can also test UDP using -u w/ iperf3 and adjust the bandwidth w/
> > -b 300m (or other Mbps)...
> >
> > > root@nanopi-r1s-h5:~ # iperf3 -c 192.168.111.1
> > > Connecting to host 192.168.111.1, port 5201
> > > [  5] local 192.168.111.10 port 28569 connected to 192.168.111.1 port
> > 5201
> > > [ ID] Interval   Transfer Bitrate Retr  Cwnd
> > > [  5]   0.00-1.00   sec  27.4 MBytes   230 Mbits/sec0   95.4 KBytes
> > > [  5]   1.00-2.00   sec  27.6 MBytes   232 Mbits/sec0   95.4 KBytes
> > > [  5]   2.00-3.00   sec  27.7 MBytes   232 Mbits/sec0   95.4 KBytes
> > > [  5]   3.00-4.00   sec  27.6 MBytes   232 Mbits/sec0   95.4 KBytes
> > > [  5]   4.00-5.00   sec  27.6 MBytes   232 Mbits/sec0   95.4 KBytes
> > > [  5]   5.00-6.00   sec  27.6 MBytes   232 Mbits/sec0   95.4 KBytes
> > > [  5]   6.00-7.00   sec  27.7 MBytes   232 Mbits/sec0   95.4 KBytes
> > > [  5]   7.00-8.00   sec  27.7 MBytes   232 Mbits/sec0   95.4 KBytes
> > > [  5]   8.00-9.00   sec  27.6 MBytes   232 Mbits/sec0   95.4 KBytes
> > > [  5]   9.00-10.00  sec  27.6 MBytes   232 Mbits/sec0   95.4 KBytes
> > > - - - - - - - - - - - - - - - - - - - - - - - - -
> > > [ ID] Interval   Transfer Bitrate Retr
> > > [  5]   0.00-10.00  sec   276 MBytes   232 Mbits/sec0
> >  sender
> > > [  5]   0.00-10.79  sec   276 MBytes   215 Mbits/sec
> > >  receiver
> > >
> > > iperf Done.
> > > root@nanopi-r1s-h5:~ # iperf3 -c 192.168.111.1 -R
> > > Connecting to host 192.168.111.1, port 5201
> > > Reverse mode, remote host 192.168.111.1 is sending
> > > [  5] local 192.168.111.10 port 29384 connected to 192.168.111.1 port
> > 5201
> > > [ ID] Interval   Transfer Bitrate
> > > [  5]   0.00-1.00   sec  12.1 MBytes   102 Mbits/sec
> > > [  5]   1.00-2.00   sec  12.1 MBytes   102 Mbits/sec
> > > [  5]   2.00-3.00   sec  12.1 MBytes   101 Mbits/sec
> > > [  5]   3.00-4.00   sec  12.1 MBytes   102 Mbits/sec
> > > [  5]   4.00-5.00   sec  12.1 MBytes   102 Mbits/sec
> > > [  5]   5.00-6.00   sec  12.1 MBytes   102 Mbits/sec
> > > [  5]   6.00-7.00   sec  12.1 MBytes   101 Mbits/sec
> > > [  5]   7.00-8.00   sec  12.1 MBytes   102 Mbits/sec
> > > [  5]   8.00-9.00   sec  12.1 MBytes   101 Mbits/sec
> > > [  5]   9.00-10.00  sec  12.1 MBytes   102 Mbits/sec
> > > - - - - - - - - - - - - - - - - - - - - - - - - -
> >

Re: Problem compiling chromium

2020-08-09 Thread John-Mark Gurney
Filippo Moretti wrote this message on Sun, Aug 02, 2020 at 08:23 +:
> This is the last eror I get attempting to compile chromium on amd64 arch 
> R363570[263/38159] python ../../mojo/public/tools/mojom/mojom_parser.py 
> --input-root /usr/ports/www/chromium/work/chromium-84.0.4147.105/ 
> --input-root 
> /usr/ports/www/chromium/work/chromium-84.0.4147.105/out/Release/gen 
> --output-root 
> /usr/ports/www/chromium/work/chromium-84.0.4147.105/out/Release/gen 
> --mojom-file-list=__mojo_public_mojom_base_base__parser___build_toolchain_linux_clang_x64__rule..rsp
>  --check-imports 
> /usr/ports/www/chromium/work/chromium-84.0.4147.105/out/Release/gen/mojo/public/mojom/base/base.build_metadata
>  --enable-feature file_path_is_string --enable-feature is_posix 
> --enable-feature is_linux
> [264/38159] python ../../tools/licenses.py --target-os=freebsd --depfile 
> gen/components/resources/about_credits.d credits 
> gen/components/resources/about_credits.html
> FAILED: gen/components/resources/about_credits.html 
> python ../../tools/licenses.py --target-os=freebsd --depfile 
> gen/components/resources/about_credits.d credits 
> gen/components/resources/about_credits.html
> Traceback (most recent call last):
>   File "../../tools/licenses.py", line 807, in 
>     sys.exit(main())
>   File "../../tools/licenses.py", line 792, in main
>     args.gn_target, args.depfile): File "../../tools/licenses.py", line 720, 
> in GenerateCredits
>     license_file_list + ['build.ninja'])
>   File 
> "/usr/ports/www/chromium/work/chromium-84.0.4147.105/build/android/gyp/util/build_utils.py",
>  line 619, in WriteDepfile
>     inputs = ComputePythonDependencies() + inputs
>   File 
> "/usr/ports/www/chromium/work/chromium-84.0.4147.105/build/android/gyp/util/build_utils.py",
>  line 557, in ComputePythonDependencies
>     p for p in abs_module_paths if p.startswith(abs_dir_source_root)
>   File 
> "/usr/ports/www/chromium/work/chromium-84.0.4147.105/build/android/gyp/util/build_utils.py",
>  line 557, in 
>     p for p in abs_module_paths if p.startswith(abs_dir_source_root)
>   File 
> "/usr/ports/www/chromium/work/.bin/../../../../../local/lib/python3.7/posixpath.py",
>  line 378, in abspath
>     path = os.fspath(path)
> TypeError: expected str, bytes or os.PathLike object, not NoneType
> ninja: build stopped: subcommand failed.
> *** Error code 1
> 
> Stop.
> make[1]: stopped in /usr/ports/www/chromium

Are you sure that chromium is Python 3 clean?  Python 2 was recently
deprecated, so likely this is fallout from that.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: CFT: major update to if_ure (patch did not apply cleanly for head -r363510) (example PowerMac problem?)

2020-07-27 Thread John-Mark Gurney
Mark Millard wrote this message on Mon, Jul 27, 2020 at 17:15 -0700:
> On 2020-Jul-27, at 16:43, Mark Millard  wrote:
> 
> > On 2020-Jul-26, at 18:20, John-Mark Gurney  wrote:
> > 
> >> Mark Millard wrote this message on Sat, Jul 25, 2020 at 19:13 -0700:
> >>> For reference for what applying the patch
> >>> reported (see Hunk #14):
> >> 
> >> Ok, updated it to be relative to r363583...
> >> 
> >> I had made a white spcae commit to if_ure.c, but hadn't made the
> >> patch relative to it after that commit.. should work now..
> > 
> > I updated an old PowerMac G5 (2 sockets/2 cores each) to
> > head -r363590 with the update patch and tjen plugged in the
> > USB EtherNet device. The result (extracted from dmesg -a)
> > was:
> > 
> > usb_alloc_device: set address 2 failed (USB_ERR_TIMEOUT, ignored)
> > usbd_setup_device_desc: getting device descriptor at addr 2 failed, 
> > USB_ERR_TIMEOUT
> > usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_TIMEOUT, 
> > ignored)
> > usbd_setup_device_desc: getting device descriptor at addr 2 failed, 
> > USB_ERR_TIMEOUT
> > usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_TIMEOUT, 
> > ignored)
> > usbd_setup_device_desc: getting device descriptor at addr 2 failed, 
> > USB_ERR_TIMEOUT
> > usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_TIMEOUT, 
> > ignored)
> > usbd_setup_device_desc: getting device descriptor at addr 2 failed, 
> > USB_ERR_TIMEOUT
> > usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_TIMEOUT, 
> > ignored)
> > usbd_setup_device_desc: getting device descriptor at addr 2 failed, 
> > USB_ERR_TIMEOUT
> > ugen2.2:  at usbus2 (disconnected)
> > uhub_reattach_port: could not allocate new device
> > 
> > Unfortunately, I'd not tried a PowerMac with the type of
> > device before the update. I do not know if the above is
> > new behavior or not.
> > 
> > The PowerMac is big-endian, which is what got me to think
> > about trying it there. The PowerMac is also 64-bit running
> > a 64-bit FreeBSD. Its USB is 2.0.
> > 
> > (It also has 2 GigaBit EtherNet ports of its own so I'm not
> > likely to use a USB device outside special testing.)
> > 
> 
> I tried what normally shows as an axge0, but
> trying on the PowerMac G5. It got the same sort
> of messages as above. The problem does not seem
> to be tied to your patch.
> 
> It does prevent my testing the patch on the G5.

Yeah, I was going to say that the above messages are before any of
may changes get run, so it's unlikely a problem w/ my patch...
If the USB device can't get an address on the bus, then it can't
even ask what type of device it is to load the driver.

Thanks for trying though, maybe someone on the -powerpc list knows
of a fix for that.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: CFT: major update to if_ure

2020-07-27 Thread John-Mark Gurney
Ganbold Tsagaankhuu wrote this message on Mon, Jul 27, 2020 at 18:29 +0800:
> On Mon, Jul 27, 2020 at 5:14 AM John-Mark Gurney  wrote:
> 
> > Ganbold Tsagaankhuu wrote this message on Sun, Jul 26, 2020 at 11:05 +0800:
> > > On Sun, Jul 26, 2020 at 7:13 AM John-Mark Gurney 
> > wrote:
> > >
> > > > Hello,
> > > >
> > > > I'd like people who have ure (RealTek) based USB devices to test
> > > > review D25809[0].
> > > >
> > > > This update adds support for:
> > > > - HW VLAN tagging
> > > > - HW checksum offload for IPv4 and IPv6
> > > > - tx and rx aggreegation (for full gige speeds)
> > > > - multiple transactions
> > > >
> > > > In my testing, I am able to get 900-950Mbps depending upon
> > > > TCP or UDP, which is a significant improvement over the previous
> > > > 91Mbps (~8kint/sec*1500bytes/packet*1packet/int).
> > >
> > > Does performance improve for if_ure device on USB2?
> > > I will try to test it in a couple of days on NanoPI R1 and R1S boards.
> >
> > Yes, it should.
> >
> > I never tested the before driver on USB2, but I'm now able to get
> > 211Mbps TX and 190Mbps RX TCP, and 227Mbps TX and 225Mbps RX UDP.
> >
> > I believe it is likely that the same 91Mbps speed limit applied to
> > USB2 as well.
> 
> Couldn't find your iperf test scripts and I tested only tcp:

My test script isn't performance, just features, and I'm thinking about
how/where to publish it...

You can also test UDP using -u w/ iperf3 and adjust the bandwidth w/
-b 300m (or other Mbps)...

> root@nanopi-r1s-h5:~ # iperf3 -c 192.168.111.1
> Connecting to host 192.168.111.1, port 5201
> [  5] local 192.168.111.10 port 28569 connected to 192.168.111.1 port 5201
> [ ID] Interval   Transfer Bitrate Retr  Cwnd
> [  5]   0.00-1.00   sec  27.4 MBytes   230 Mbits/sec0   95.4 KBytes
> [  5]   1.00-2.00   sec  27.6 MBytes   232 Mbits/sec0   95.4 KBytes
> [  5]   2.00-3.00   sec  27.7 MBytes   232 Mbits/sec0   95.4 KBytes
> [  5]   3.00-4.00   sec  27.6 MBytes   232 Mbits/sec0   95.4 KBytes
> [  5]   4.00-5.00   sec  27.6 MBytes   232 Mbits/sec0   95.4 KBytes
> [  5]   5.00-6.00   sec  27.6 MBytes   232 Mbits/sec0   95.4 KBytes
> [  5]   6.00-7.00   sec  27.7 MBytes   232 Mbits/sec0   95.4 KBytes
> [  5]   7.00-8.00   sec  27.7 MBytes   232 Mbits/sec0   95.4 KBytes
> [  5]   8.00-9.00   sec  27.6 MBytes   232 Mbits/sec0   95.4 KBytes
> [  5]   9.00-10.00  sec  27.6 MBytes   232 Mbits/sec0   95.4 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval   Transfer Bitrate Retr
> [  5]   0.00-10.00  sec   276 MBytes   232 Mbits/sec0 sender
> [  5]   0.00-10.79  sec   276 MBytes   215 Mbits/sec
>  receiver
> 
> iperf Done.
> root@nanopi-r1s-h5:~ # iperf3 -c 192.168.111.1 -R
> Connecting to host 192.168.111.1, port 5201
> Reverse mode, remote host 192.168.111.1 is sending
> [  5] local 192.168.111.10 port 29384 connected to 192.168.111.1 port 5201
> [ ID] Interval   Transfer Bitrate
> [  5]   0.00-1.00   sec  12.1 MBytes   102 Mbits/sec
> [  5]   1.00-2.00   sec  12.1 MBytes   102 Mbits/sec
> [  5]   2.00-3.00   sec  12.1 MBytes   101 Mbits/sec
> [  5]   3.00-4.00   sec  12.1 MBytes   102 Mbits/sec
> [  5]   4.00-5.00   sec  12.1 MBytes   102 Mbits/sec
> [  5]   5.00-6.00   sec  12.1 MBytes   102 Mbits/sec
> [  5]   6.00-7.00   sec  12.1 MBytes   101 Mbits/sec
> [  5]   7.00-8.00   sec  12.1 MBytes   102 Mbits/sec
> [  5]   8.00-9.00   sec  12.1 MBytes   101 Mbits/sec
> [  5]   9.00-10.00  sec  12.1 MBytes   102 Mbits/sec
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval   Transfer Bitrate Retr
> [  5]   0.00-11.25  sec   121 MBytes  90.3 Mbits/sec  2539
> sender
> [  5]   0.00-10.00  sec   121 MBytes   101 Mbits/sec
>  receiver
> 
> iperf Done.
> root@nanopi-r1s-h5:~ # sysctl -a | grep cpu.0.freq
> dev.cpu.0.freq_levels: 1248/-1 1008/-1 816/-1 624/-1 480/-1
> dev.cpu.0.freq: 1248

Hmmm... The reverse seems slow, but I can't think of why it'd be that
slow though.  When I did my tests on the USB2 ports, both directions
were about the same speed...

Thanks for the test!  Great to hear things are working...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: CFT: major update to if_ure (patch did not apply cleanly for head -r363510)

2020-07-26 Thread John-Mark Gurney
Mark Millard wrote this message on Sat, Jul 25, 2020 at 19:13 -0700:
> For reference for what applying the patch
> reported (see Hunk #14):

Ok, updated it to be relative to r363583...

I had made a white spcae commit to if_ure.c, but hadn't made the
patch relative to it after that commit.. should work now..

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: CFT: major update to if_ure

2020-07-26 Thread John-Mark Gurney
Ganbold Tsagaankhuu wrote this message on Sun, Jul 26, 2020 at 11:05 +0800:
> On Sun, Jul 26, 2020 at 7:13 AM John-Mark Gurney  wrote:
> 
> > Hello,
> >
> > I'd like people who have ure (RealTek) based USB devices to test
> > review D25809[0].
> >
> > This update adds support for:
> > - HW VLAN tagging
> > - HW checksum offload for IPv4 and IPv6
> > - tx and rx aggreegation (for full gige speeds)
> > - multiple transactions
> >
> > In my testing, I am able to get 900-950Mbps depending upon
> > TCP or UDP, which is a significant improvement over the previous
> > 91Mbps (~8kint/sec*1500bytes/packet*1packet/int).
> 
> Does performance improve for if_ure device on USB2?
> I will try to test it in a couple of days on NanoPI R1 and R1S boards.

Yes, it should.

I never tested the before driver on USB2, but I'm now able to get
211Mbps TX and 190Mbps RX TCP, and 227Mbps TX and 225Mbps RX UDP.

I believe it is likely that the same 91Mbps speed limit applied to
USB2 as well.

> > [0] https://reviews.freebsd.org/D25809

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


CFT: major update to if_ure

2020-07-25 Thread John-Mark Gurney
Hello,

I'd like people who have ure (RealTek) based USB devices to test
review D25809[0].

This update adds support for:
- HW VLAN tagging
- HW checksum offload for IPv4 and IPv6
- tx and rx aggreegation (for full gige speeds)
- multiple transactions

In my testing, I am able to get 900-950Mbps depending upon
TCP or UDP, which is a significant improvement over the previous
91Mbps (~8kint/sec*1500bytes/packet*1packet/int).

Thanks.

[0] https://reviews.freebsd.org/D25809

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: somewhat reproducable vimage panic

2020-07-25 Thread John-Mark Gurney
John-Mark Gurney wrote this message on Thu, Jul 23, 2020 at 16:49 -0700:
> Kristof Provost wrote this message on Thu, Jul 23, 2020 at 11:02 +0200:
> > On 23 Jul 2020, at 11:00, Bjoern A. Zeeb wrote:
> > > On 23 Jul 2020, at 8:09, Kristof Provost wrote:
> > >
> > >> On 23 Jul 2020, at 9:19, Kristof Provost wrote:
> > >>> On 23 Jul 2020, at 0:15, John-Mark Gurney wrote:
> > >>>> So, it's pretty easy to trigger, just attach a couple USB ethernet
> > >>>> adapters, in my case, they were ure, but likely any two spare 
> > >>>> ethernet
> > >>>> interfaces will work, and wire them back to back..
> > >>>>
> > >>> I???ve been able to trigger it using epair as well:
> > >>>
> > >>> `sudo sh testinterfaces.txt epair0a epair0b`
> > >>>
> > >>> I did have to comment out the waitcarrier() check.
> > >>>
> > >> I???ve done a little bit of digging, and I think I???m starting to 
> > >> see how this breaks.
> > >>
> > >> This always affects the jailed vlan interfaces. They???re getting 
> > >> deleted, but the ifp doesn???t go away just yet because it???s still 
> > >> in use by the multicast code.
> > >> The multicast code does its cleanup in task queues,
> > >
> > > Wow, did I miss that back then? Did I review a change and not notice? 
> > > Sorry if that was the case.
> > >
> > > Vnet teardown is blocking and forceful.
> > > Doing deferred cleanup work isn???t a good idea at all.
> > > I think that is the real problem here.
> > >
> > > I???d rather have us fix this than putting more bandaids into the 
> > > code.
> > >
> > Yeah, agreed. I think hselasky has a better fix: 
> > https://reviews.freebsd.org/D24914
> > 
> > I just saw his e-mail in a different thread.
> 
> I'm testing out this patch now, and let people know how it goes.. It'll
> be nice to not have to worry about these panics..

So far so good...  I am getting these on occasion:
in6_purgeaddr: err=65, destination address delete failed

But that's more that the patch prevented a panic.

The other issue that I'm now seeing is that because we don't forcefully
clear out the multicast task, it can take a good 20+ seconds from the
time a jail is destroyed to the interface appearing again in vnet0.
Pretty sure this is related to the dmesg from above...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: somewhat reproducable vimage panic

2020-07-23 Thread John-Mark Gurney
Kristof Provost wrote this message on Thu, Jul 23, 2020 at 11:02 +0200:
> On 23 Jul 2020, at 11:00, Bjoern A. Zeeb wrote:
> > On 23 Jul 2020, at 8:09, Kristof Provost wrote:
> >
> >> On 23 Jul 2020, at 9:19, Kristof Provost wrote:
> >>> On 23 Jul 2020, at 0:15, John-Mark Gurney wrote:
> >>>> So, it's pretty easy to trigger, just attach a couple USB ethernet
> >>>> adapters, in my case, they were ure, but likely any two spare 
> >>>> ethernet
> >>>> interfaces will work, and wire them back to back..
> >>>>
> >>> I???ve been able to trigger it using epair as well:
> >>>
> >>> `sudo sh testinterfaces.txt epair0a epair0b`
> >>>
> >>> I did have to comment out the waitcarrier() check.
> >>>
> >> I???ve done a little bit of digging, and I think I???m starting to 
> >> see how this breaks.
> >>
> >> This always affects the jailed vlan interfaces. They???re getting 
> >> deleted, but the ifp doesn???t go away just yet because it???s still 
> >> in use by the multicast code.
> >> The multicast code does its cleanup in task queues,
> >
> > Wow, did I miss that back then? Did I review a change and not notice? 
> > Sorry if that was the case.
> >
> > Vnet teardown is blocking and forceful.
> > Doing deferred cleanup work isn???t a good idea at all.
> > I think that is the real problem here.
> >
> > I???d rather have us fix this than putting more bandaids into the 
> > code.
> >
> Yeah, agreed. I think hselasky has a better fix: 
> https://reviews.freebsd.org/D24914
> 
> I just saw his e-mail in a different thread.

I'm testing out this patch now, and let people know how it goes.. It'll
be nice to not have to worry about these panics..

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: somewhat reproducable vimage panic

2020-07-22 Thread John-Mark Gurney
Bjoern A. Zeeb wrote this message on Wed, Jul 22, 2020 at 20:43 +:
> On 22 Jul 2020, at 19:34, John-Mark Gurney wrote:
> 
> > John-Mark Gurney wrote this message on Tue, Jul 21, 2020 at 23:05 
> > -0700:
> >> Peter Libassi wrote this message on Wed, Jul 22, 2020 at 06:54 +0200:
> >>> Is this related to
> >>>
> >>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=234985 
> >>> <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=234985> and 
> >>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=238326 
> >>> <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=238326>
> >>
> >> Definitely not 234985..  I'm using ue interfaces, and so they don't
> >> get destroyed while the jail is going away...
> >>
> >> I don't think it's 238326 either.  This is 100% reliable and it's in
> >> the IP multicast code..  It looks like in_multi isn't holding an
> >> interface or address lock waiting for things to free up...
> >
> > Did a little more poking, and it looks like the vnet is free'd before
> > the ifnet is free'd causing this problem:
> > (kgdb) print inm->inm_ifp[0].if_refcount
> > $5 = 1
> > (kgdb) print inm->inm_ifp[0].if_vnet[0]
> > $6 = {vnet_le = {le_next = 0xdeadc0dedeadc0de, le_prev = 
> > 0xdeadc0dedeadc0de},
> >   vnet_magic_n = 3735929054, vnet_ifcnt = 3735929054,
> >   vnet_sockcnt = 3735929054, vnet_state = 3735929054,
> >   vnet_data_mem = 0xdeadc0dedeadc0de, vnet_data_base = 
> > 16045693110842147038,
> >   vnet_shutdown = 222}
> >
> > So the multicast code is fine, it holds and releases a reference to
> > ifnet..
> >
> > The issue is that the reference to the ifnet doesn't involve a
> > reference to the vnet/prison.
> 
> Does it need to?  The ifnet cannot go away while something holds a 
> reference to it, right?

It's the other way around that's the problem.. the ifnet is holding an
invalid vnet pointer that got free'd.

Maybe the problem isn't the tear down, but that the vnet pointer isn't
changed/restored before the free?

> Sounds more like the teardown order is wrong (again)?
> 
> There should be no more multicast when IP etc. is gone.  That means MC 
> doesn???t properly cleanup itself.

Don't know, just know that it's easy to trigger right now...  I haven't
tested on prior releases, but if you'd like me to, it isn't too hard for
me to test...

> I guess I should go back now and re-read your original problem statement 
> on how you trigger this..

So, it's pretty easy to trigger, just attach a couple USB ethernet
adapters, in my case, they were ure, but likely any two spare ethernet
interfaces will work, and wire them back to back..

Run the script attached earlier in the thread, providing it the name
of the two interfaces as arguments, and run it a few times.  You might
get failures or not.  It shouldn't matter.  After a few runs, it'll
panic...

I just tested this (to make sure my ure changes weren't causing addition
problems) using
FreeBSD-13.0-CURRENT-amd64-20200625-r362596-memstick.img.xz, so it's
stock reproducable.

Thanks for looking into this!

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: somewhat reproducable vimage panic

2020-07-22 Thread John-Mark Gurney
John-Mark Gurney wrote this message on Tue, Jul 21, 2020 at 23:05 -0700:
> Peter Libassi wrote this message on Wed, Jul 22, 2020 at 06:54 +0200:
> > Is this related to 
> > 
> > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=234985 
> > <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=234985> and 
> > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=238326 
> > <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=238326>
> 
> Definitely not 234985..  I'm using ue interfaces, and so they don't
> get destroyed while the jail is going away...
> 
> I don't think it's 238326 either.  This is 100% reliable and it's in
> the IP multicast code..  It looks like in_multi isn't holding an
> interface or address lock waiting for things to free up...

Did a little more poking, and it looks like the vnet is free'd before
the ifnet is free'd causing this problem:
(kgdb) print inm->inm_ifp[0].if_refcount 
$5 = 1
(kgdb) print inm->inm_ifp[0].if_vnet[0]  
$6 = {vnet_le = {le_next = 0xdeadc0dedeadc0de, le_prev = 0xdeadc0dedeadc0de},
  vnet_magic_n = 3735929054, vnet_ifcnt = 3735929054,
  vnet_sockcnt = 3735929054, vnet_state = 3735929054,
  vnet_data_mem = 0xdeadc0dedeadc0de, vnet_data_base = 16045693110842147038,
  vnet_shutdown = 222}

So the multicast code is fine, it holds and releases a reference to
ifnet..

The issue is that the reference to the ifnet doesn't involve a
reference to the vnet/prison.

I have noticed that a number of times after a jail destroy that one
of my interfaces doesn't make it back to the main OS.. it's just gone..

I can "make" it reappear by reseting the hardware, but that does imply
that an ifnet is hanging out in limbo...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: somewhat reproducable vimage panic

2020-07-22 Thread John-Mark Gurney
Peter Libassi wrote this message on Wed, Jul 22, 2020 at 06:54 +0200:
> Is this related to 
> 
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=234985 
> <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=234985> and 
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=238326 
> <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=238326>

Definitely not 234985..  I'm using ue interfaces, and so they don't
get destroyed while the jail is going away...

I don't think it's 238326 either.  This is 100% reliable and it's in
the IP multicast code..  It looks like in_multi isn't holding an
interface or address lock waiting for things to free up...

> > 21 juli 2020 kl. 22:23 skrev John-Mark Gurney :
> > 
> > Marko Zec wrote this message on Tue, Jul 21, 2020 at 11:31 +0200:
> >> On Tue, 21 Jul 2020 02:16:55 -0700
> >> John-Mark Gurney  wrote:
> >> 
> >>> I'm running:
> >>> FreeBSD test 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r362596: Thu Jun 25
> >>> 05:02:51 UTC 2020
> >>> r...@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC
> >>> amd64
> >>> 
> >>> and I'm working on improve the if_ure driver.  I've put together a
> >>> little script that I've attached that I'm using to test the driver..
> >>> It puts a couple ue interfaces each into their own jail, configures
> >>> them, and tries to pass traffic.  This assumes that the two interfaces
> >>> are connected together.
> >>> 
> >>> Pretty regularly when destroying the jails, I get the following
> >>> panic: CURVNET_SET at /usr/src/sys/netinet/in_mcast.c:626
> >>> inm_release() curvnet=0 vnet=0xf80154c82a80
> >> 
> >> Perhaps the attached patch could help? (disclaimer: not even
> >> compile-tested)
> > 
> > The patch compiled, but it just moved the panic earlier than before.
> > 
> > #4  0x80bc2123 in panic (fmt=)
> >at ../../../kern/kern_shutdown.c:839
> > #5  0x80d61726 in inm_release_task (arg=, 
> >pending=) at ../../../netinet/in_mcast.c:633
> > #6  0x80c2166a in taskqueue_run_locked (queue=0xf800033cfd00)
> >at ../../../kern/subr_taskqueue.c:476
> > #7  0x80c226e4 in taskqueue_thread_loop (arg=)
> >at ../../../kern/subr_taskqueue.c:793
> > 
> > Now it panics at the location of the new CURVNET_SET and not the
> > old one..
> > 
> > Ok, decided to dump the contents of the vnet, and it looks like
> > it's a use after free:
> > (kgdb) print/x *(struct vnet *)0xf8012a283140
> > $2 = {vnet_le = {le_next = 0xdeadc0dedeadc0de, le_prev = 
> > 0xdeadc0dedeadc0de}, vnet_magic_n = 0xdeadc0de, 
> >  vnet_ifcnt = 0xdeadc0de, vnet_sockcnt = 0xdeadc0de, vnet_state = 
> > 0xdeadc0de, vnet_data_mem = 0xdeadc0dedeadc0de, 
> >  vnet_data_base = 0xdeadc0dedeadc0de, vnet_shutdown = 0xde}
> > 
> > The patch did seem to make it happen quicker, or maybe I was just more
> > lucky this morning...
> > 
> >>> (kgdb) #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
> >>> #1  doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:394
> >>> #2  0x80bc6250 in kern_reboot (howto=260)
> >>>at /usr/src/sys/kern/kern_shutdown.c:481
> >>> #3  0x80bc66aa in vpanic (fmt=, ap= >>> out>) at /usr/src/sys/kern/kern_shutdown.c:913
> >>> #4  0x80bc6403 in panic (fmt=)
> >>>at /usr/src/sys/kern/kern_shutdown.c:839
> >>> #5  0x80d6553b in inm_release (inm=0xf80029043700)
> >>>at /usr/src/sys/netinet/in_mcast.c:630
> >>> #6  inm_release_task (arg=, pending=)
> >>>at /usr/src/sys/netinet/in_mcast.c:312
> >>> #7  0x80c2521a in taskqueue_run_locked
> >>> (queue=0xf80003116b00) at /usr/src/sys/kern/subr_taskqueue.c:476
> >>> #8  0x80c26294 in taskqueue_thread_loop (arg=)
> >>>at /usr/src/sys/kern/subr_taskqueue.c:793
> >>> #9  0x80b830f0 in fork_exit (
> >>>callout=0x80c26200 , 
> >>>arg=0x81cf4f70 ,
> >>> frame=0xfe0049e99b80) at /usr/src/sys/kern/kern_fork.c:1052
> >>> #10 
> >>> (kgdb) 
> >>> 
> >>> I have the core files so I can get additional information.
> >>> 
> >>> Let me know if you need any additional information.
> >>> 
> >> 
> > 
> >> Index: sys/netinet/in_mcast.c
> >> 

Re: somewhat reproducable vimage panic

2020-07-21 Thread John-Mark Gurney
Marko Zec wrote this message on Tue, Jul 21, 2020 at 11:31 +0200:
> On Tue, 21 Jul 2020 02:16:55 -0700
> John-Mark Gurney  wrote:
> 
> > I'm running:
> > FreeBSD test 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r362596: Thu Jun 25
> > 05:02:51 UTC 2020
> > r...@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC
> >  amd64
> > 
> > and I'm working on improve the if_ure driver.  I've put together a
> > little script that I've attached that I'm using to test the driver..
> > It puts a couple ue interfaces each into their own jail, configures
> > them, and tries to pass traffic.  This assumes that the two interfaces
> > are connected together.
> > 
> > Pretty regularly when destroying the jails, I get the following
> > panic: CURVNET_SET at /usr/src/sys/netinet/in_mcast.c:626
> > inm_release() curvnet=0 vnet=0xf80154c82a80
> 
> Perhaps the attached patch could help? (disclaimer: not even
> compile-tested)

The patch compiled, but it just moved the panic earlier than before.

#4  0x80bc2123 in panic (fmt=)
at ../../../kern/kern_shutdown.c:839
#5  0x80d61726 in inm_release_task (arg=, 
pending=) at ../../../netinet/in_mcast.c:633
#6  0x80c2166a in taskqueue_run_locked (queue=0xf800033cfd00)
at ../../../kern/subr_taskqueue.c:476
#7  0x80c226e4 in taskqueue_thread_loop (arg=)
at ../../../kern/subr_taskqueue.c:793

Now it panics at the location of the new CURVNET_SET and not the
old one..

Ok, decided to dump the contents of the vnet, and it looks like
it's a use after free:
(kgdb) print/x *(struct vnet *)0xf8012a283140
$2 = {vnet_le = {le_next = 0xdeadc0dedeadc0de, le_prev = 0xdeadc0dedeadc0de}, 
vnet_magic_n = 0xdeadc0de, 
  vnet_ifcnt = 0xdeadc0de, vnet_sockcnt = 0xdeadc0de, vnet_state = 0xdeadc0de, 
vnet_data_mem = 0xdeadc0dedeadc0de, 
  vnet_data_base = 0xdeadc0dedeadc0de, vnet_shutdown = 0xde}

The patch did seem to make it happen quicker, or maybe I was just more
lucky this morning...

> > (kgdb) #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
> > #1  doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:394
> > #2  0x80bc6250 in kern_reboot (howto=260)
> > at /usr/src/sys/kern/kern_shutdown.c:481
> > #3  0x80bc66aa in vpanic (fmt=, ap= > out>) at /usr/src/sys/kern/kern_shutdown.c:913
> > #4  0x80bc6403 in panic (fmt=)
> > at /usr/src/sys/kern/kern_shutdown.c:839
> > #5  0x80d6553b in inm_release (inm=0xf80029043700)
> > at /usr/src/sys/netinet/in_mcast.c:630
> > #6  inm_release_task (arg=, pending=)
> > at /usr/src/sys/netinet/in_mcast.c:312
> > #7  0x80c2521a in taskqueue_run_locked
> > (queue=0xf80003116b00) at /usr/src/sys/kern/subr_taskqueue.c:476
> > #8  0x80c26294 in taskqueue_thread_loop (arg=)
> > at /usr/src/sys/kern/subr_taskqueue.c:793
> > #9  0x80b830f0 in fork_exit (
> > callout=0x80c26200 , 
> > arg=0x81cf4f70 ,
> > frame=0xfe0049e99b80) at /usr/src/sys/kern/kern_fork.c:1052
> > #10 
> > (kgdb) 
> > 
> > I have the core files so I can get additional information.
> > 
> > Let me know if you need any additional information.
> > 
> 

> Index: sys/netinet/in_mcast.c
> ===
> --- sys/netinet/in_mcast.c(revision 363386)
> +++ sys/netinet/in_mcast.c(working copy)
> @@ -309,8 +309,10 @@
>   IN_MULTI_LOCK();
>   SLIST_FOREACH_SAFE(inm, _free_tmp, inm_nrele, tinm) {
>   SLIST_REMOVE_HEAD(_free_tmp, inm_nrele);
> + CURVNET_SET(inm->inm_ifp->if_vnet);
>   MPASS(inm);
>   inm_release(inm);
> + CURVNET_RESTORE();
>   }
>   IN_MULTI_UNLOCK();
>  }


-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


somewhat reproducable vimage panic

2020-07-21 Thread John-Mark Gurney
I'm running:
FreeBSD test 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r362596: Thu Jun 25 05:02:51 
UTC 2020 
r...@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64

and I'm working on improve the if_ure driver.  I've put together a
little script that I've attached that I'm using to test the driver..
It puts a couple ue interfaces each into their own jail, configures
them, and tries to pass traffic.  This assumes that the two interfaces
are connected together.

Pretty regularly when destroying the jails, I get the following
panic: CURVNET_SET at /usr/src/sys/netinet/in_mcast.c:626 inm_release() 
curvnet=0 vnet=0xf80154c82a80

(kgdb) #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:394
#2  0x80bc6250 in kern_reboot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:481
#3  0x80bc66aa in vpanic (fmt=, ap=)
at /usr/src/sys/kern/kern_shutdown.c:913
#4  0x80bc6403 in panic (fmt=)
at /usr/src/sys/kern/kern_shutdown.c:839
#5  0x80d6553b in inm_release (inm=0xf80029043700)
at /usr/src/sys/netinet/in_mcast.c:630
#6  inm_release_task (arg=, pending=)
at /usr/src/sys/netinet/in_mcast.c:312
#7  0x80c2521a in taskqueue_run_locked (queue=0xf80003116b00)
at /usr/src/sys/kern/subr_taskqueue.c:476
#8  0x80c26294 in taskqueue_thread_loop (arg=)
at /usr/src/sys/kern/subr_taskqueue.c:793
#9  0x80b830f0 in fork_exit (
callout=0x80c26200 , 
arg=0x81cf4f70 , frame=0xfe0049e99b80)
at /usr/src/sys/kern/kern_fork.c:1052
#10 
(kgdb) 

I have the core files so I can get additional information.

Let me know if you need any additional information.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
#!/bin/sh -
#
# ls testinterfaces.sh | entr sh -c 'fsync testinterfaces.sh && sh 
testinterfaces.sh ue0 ue1'
#

testiface="$1"
checkiface="$2"

echo starting, test $1, check $2
# ifconfig  -m to get caps  
getcaps()
{
ifconfig -m "$1" | awk '$1 ~ /^capabilities=/ { split($0, a, "<"); 
split(a[2], b, ">"); split(b[1], caps, ","); for (i in caps) print caps[i] }'
}


if ! testjail=$(jail -i -c persist=1 path=/ name=testjail vnet=new 
vnet.interface="$testiface"); then
echo failed to start test jail
exit 1
fi

if ! checkjail=$(jail -i -c persist=1 path=/ name=checkjail vnet=new 
vnet.interface="$checkiface"); then
echo failed to start check jail
exit 1
fi

cleanup()
{
jail -r "$testjail"
jail -r "$checkjail"
}

trap cleanup EXIT

run()
{
if [ x"$1" = x"check" ]; then
jid="$checkjail"
elif [ x"$1" = x"test" ]; then
jid="$testjail"
else
echo Invalid: "$1" >&2
exit 1
fi

shift
jexec "$jid" "$@"
}

ifjail()
{
if [ x"$1" = x"check" ]; then
iface="$checkiface"
elif [ x"$1" = x"test" ]; then
iface="$testiface"
else
echo Invalid: "$1" >&2
exit 1
fi

j="$1"
shift

run "$j" ifconfig "$iface" "$@"
}

waitcarrier()
{
local i

for i in test check; do
while :; do
if ifjail "$i" | grep 1000baseT >/dev/null; then
break
fi
sleep .5
done
done
}

hwvlantest()
{
run test ifconfig lo0 up
run check ifconfig lo0 up

for i in "vlanhwtag" "-vlanhwtag"; do
ifjail test down
ifjail check down

ifjail test "$i"
ifjail check -vlanhwtag

ifjail test up
ifjail check up

sleep 2

run test ifconfig $testiface.42 create 172.30.5.5/24
run check ifconfig $checkiface.42 create 172.30.5.4/24

waitcarrier

run check tcpdump -p -q -c 2 -n -i "$checkiface" vlan 42 and 
icmp &
sleep 1
if ! run test ping -c 1 -t 2 172.30.5.4; then
echo FAILED on "$i"
exit 1
fi
done
}

hwvlantest
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


test suite for NIC features...

2020-07-20 Thread John-Mark Gurney
Has anyone compiled a script/test suite for testing various NIC
features to make sure they work/function properly?

That is, being able to run a couple interfaces back to back, and turn
off the features off on one, and make sure things like checksum offload
and the like work properly?

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: slow USB 3.0 on -current

2020-07-13 Thread John-Mark Gurney
Hans Petter Selasky wrote this message on Mon, Jul 13, 2020 at 10:50 +0200:
> On 2020-07-13 03:02, John-Mark Gurney wrote:
> > MB means megabytes.. I would use Mbps for bits...  so, on Win10 and
> > NetBSD, I'm able to get 100 MBytes/sec on Win10/NetBSD, and FreeBSD,
> > I'm only getting a tenth the capability of gige at 9-10 MBytes/sec...
> > 
> > I'll note that fetch reports numbers of MBps, which is one of the tools
> > I've been using for testing.
> 
> Could you have a look at the traffic pattern, when using iperf?
> 
> usbdump -i usbusX -f Y
> 
> Where X and Y are the numbers after ugen .
> 
> Many of the network USB drivers are single buffered, because that's what 
> older USB host controllers support. Then the IRQ rate 8000 for USB 
> 2.0/3.0 and 1000 for USB 1.0, limits the number of packets per second.

Hmm...  now that I do the math, that sounds very likely what the problem
is:
8000 int/s * 1480 bytes/int * 8 bits/byte /1000/1000 == 94.72 Mbps

add in the delay to swap buffers...  and this is very close to the speed
that I'm seeing...

I wasn't sure what to look for, but the output is here:
https://www.funkthat.com/~jmg/FreeBSD/usb.a78/usbdump.0.4.txt.xz

Also, the output does not match what the man page says..  It implies
that OUT or IN, but I'm seeing SUBM and DONE instead, and the delimiters
between the feels don't make sense...

This was running:
gold,pts,/home/jmg,521$iperf3 -b 240m -u -c 192.168.0.80
Connecting to host 192.168.0.80, port 5201
[  5] local 192.168.0.2 port 65117 connected to 192.168.0.80 port 5201
[ ID] Interval   Transfer Bitrate Total Datagrams
[  5]   0.00-1.00   sec  28.6 MBytes   240 Mbits/sec  20531
[  5]   1.00-2.00   sec  28.6 MBytes   240 Mbits/sec  20548
[  5]   2.00-3.00   sec  28.6 MBytes   240 Mbits/sec  20550
[  5]   3.00-4.00   sec  28.6 MBytes   240 Mbits/sec  20546
[  5]   4.00-5.00   sec  28.6 MBytes   240 Mbits/sec  20551
[  5]   5.00-6.00   sec  28.6 MBytes   240 Mbits/sec  20545
[  5]   6.00-7.00   sec  28.6 MBytes   240 Mbits/sec  20548
[  5]   7.00-8.01   sec  28.1 MBytes   234 Mbits/sec  20175
[  5]   8.01-9.00   sec  29.1 MBytes   246 Mbits/sec  20921
[  5]   9.00-10.00  sec  28.6 MBytes   240 Mbits/sec  20548
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval   Transfer Bitrate JitterLost/Total 
Datagrams
[  5]   0.00-10.00  sec   286 MBytes   240 Mbits/sec  0.000 ms  0/205463 (0%)  
sender
[  5]   0.00-10.35  sec   107 MBytes  87.1 Mbits/sec  0.172 ms  128298/205436 
(62%)  receiver

Which is trying to send 240Mbps UDP traffic to the USB3 ethernet
machine, and the machine only receives the usual 91Mbps...

I have also published:
https://www.funkthat.com/~jmg/FreeBSD/usb.a78/umass.debug.txt.xz

Which shows the umass issue that I've been having where any umass
device on this system almost never attaches...  This was takes w/
hw.usb.umass.debug=-1
hw.usb.xhci.debug=17

I set those sysctl's, then attached the umass device (as USB3.0 SD
card reader), waited for the five retries..

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: slow USB 3.0 on -current

2020-07-13 Thread John-Mark Gurney
Mark Millard wrote this message on Mon, Jul 13, 2020 at 00:44 -0700:
> On 2020-Jul-12, at 21:51, John-Mark Gurney  wrote:
> 
> > Mark Millard wrote this message on Sun, Jul 12, 2020 at 18:26 -0700:
> >> John-Mark Gurney jmg at funkthat.com wrote on
> >> Sat Jul 11 22:44:36 UTC 2020 :
> >> 
> >>> I'm having issues getting good ethernet performance from a USB ethernet
> >>> adapter (ure) under FreeBSD on an HP EliteDesk 705 G2 Mini[1].  It's an
> >>> AMD PRO A10-8700B based system using the AMD A78 FCH chipset.
> >>> 
> >>> Under FreeBSD -current (r362596), 12.1-R and 11.4-R, the RealTek USB
> >>> adapter only gets around 10MB/sec performance.  During the transfer,
> >>> the CPU usage is only around 3-5%, so it's definitely not CPU bound.
> >>> 
> >>> I have tested Windows 10 and NetBSD 9.0 performance, and both provide
> >>> 100MB/sec+ w/o troubles.
> >>> 
> >>> I have attached dmesg from both FreeBSD -current and NetBSD 9.0.
> >>> 
> >>> Any hints on how to fix this?
> >>> 
> >>> This may be related, but I'm also having issues w/ booting when I have
> >>> both a SD USB 2.0 card reader AND the ure plugged into USB 3.0 ports.
> >>> 
> >>> If I move the SD card reader to USB 2.0, the umass device will attach
> >>> and work.  I have also attached a clip of the dmesg from that
> >>> happening.
> >>> 
> >>> Has anyone else seen this issue?  Ideas or thoughts on how to resolve
> >>> the performance issues?
> >> 
> >> It might prove useful to use iperf3 with
> >> 
> >> # iperf3 -s
> >> 
> >> on one machine and doing
> >> 
> >> # iperf3 -c ADDR
> >> . . .
> >> # iperf3 -R -c ADDR
> >> . . .
> >> 
> >> on the other. (That last swaps the
> >> sender/receiver status.)
> >> 
> >> All 3 commands will have output. The
> >> -s one will produce output for each of
> >> the -c ones.
> >> 
> >> The outputs for the sender(s) will include Cwnd
> >> (congestion window size) information that may
> >> be relevant. It will report bit rate and
> >> retry count sampling (and overall figures).

[...]

> The "iperf3 -s" should have had output with the Cwnd
> figures for the "Reverse mode" case above (and the
> distribution for the 1381 Retr total). They might
> not match when the earlier figures that you did report
> for the non-Reverse mode.

If you can tell how the Cwnd figures would help you figure out how to
make the USB3 bus run faster, I'll spend the time to retest and give
you the numbers, but I don't see how they can...

[...]

> > As you can see, both match approximately what I measured other methods,
> > so, it's definitely not the way I measured performance.
> > 
> >> My observation would be that neither type
> >> of USB3 Ethernet adapter that I've tried
> > 
> > What is the chipset that you tried?  One of the earlier ones that I
> > tried was an axe iirc, and was  limited to around 500Mbps or so...
> 
> Hmm. I only seem to be able to find one type. Its been a
> while since I've used the other and I do not know where
> it is at. For what I found:
> 
> ugen0.2:  at usbus0
> axge0 on uhub0
> axge0:  on usbus0
> miibus1:  on axge0
> rgephy0:  PHY 3 on miibus1
> rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 
> 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 
> 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
> 
> (I have access to more than one instance of the above.)

Yeah, these are the ones that are known to not be able to get close
to gige speeds, unlike the RealTek one that I am using now...

I forgot that axge is the gigabit version of axe...

> The iperf3 output that I reported was for using
> of of the above. Note that when the USB3 EtherNet
> was reciveing Cwnd was reported as 29.8 KBytes
> or smaller for the example run, much like your
> output reporting 34.4 KBytes or less for the
> example run.

They grew to match the speeds that the link could do..

> This may suggest some common constraint across various
> USB3 EtherNet devices. The Cwnd figures are probably
> too small to get near 900 Mbit/s+.

As you can see, the NetBSD results was able to grow the
Cwnd large enough to obtain performance...

The stats that I provided were from the non-USB3 machine, and for
tx'ing to matches the issue I raised in the original post... I could
provide the Cwnd, but I don't see how that will

Re: slow USB 3.0 on -current

2020-07-12 Thread John-Mark Gurney
Mark Millard wrote this message on Sun, Jul 12, 2020 at 18:26 -0700:
> John-Mark Gurney jmg at funkthat.com wrote on
> Sat Jul 11 22:44:36 UTC 2020 :
> 
> > I'm having issues getting good ethernet performance from a USB ethernet
> > adapter (ure) under FreeBSD on an HP EliteDesk 705 G2 Mini[1].  It's an
> > AMD PRO A10-8700B based system using the AMD A78 FCH chipset.
> > 
> > Under FreeBSD -current (r362596), 12.1-R and 11.4-R, the RealTek USB
> > adapter only gets around 10MB/sec performance.  During the transfer,
> > the CPU usage is only around 3-5%, so it's definitely not CPU bound.
> > 
> > I have tested Windows 10 and NetBSD 9.0 performance, and both provide
> > 100MB/sec+ w/o troubles.
> > 
> > I have attached dmesg from both FreeBSD -current and NetBSD 9.0.
> > 
> > Any hints on how to fix this?
> > 
> > This may be related, but I'm also having issues w/ booting when I have
> > both a SD USB 2.0 card reader AND the ure plugged into USB 3.0 ports.
> > 
> > If I move the SD card reader to USB 2.0, the umass device will attach
> > and work.  I have also attached a clip of the dmesg from that
> > happening.
> > 
> > Has anyone else seen this issue?  Ideas or thoughts on how to resolve
> > the performance issues?
> 
> It might prove useful to use iperf3 with
> 
> # iperf3 -s
> 
> on one machine and doing
> 
> # iperf3 -c ADDR
> . . .
> # iperf3 -R -c ADDR
> . . .
> 
> on the other. (That last swaps the
> sender/receiver status.)
> 
> All 3 commands will have output. The
> -s one will produce output for each of
> the -c ones.
> 
> The outputs for the sender(s) will include Cwnd
> (congestion window size) information that may
> be relevant. It will report bit rate and
> retry count sampling (and overall figures).

Here is the results for FreeBSD w/ USB3 ure.  .80 is the USB3
adapter side:
gold,pts,/home/jmg,502$iperf3 -c 192.168.0.80
Connecting to host 192.168.0.80, port 5201
[  5] local 192.168.0.2 port 50042 connected to 192.168.0.80 port 5201
[ ID] Interval   Transfer Bitrate Retr  Cwnd
[  5]   0.00-1.00   sec  8.94 MBytes  75.0 Mbits/sec  931   15.5 KBytes
[  5]   1.00-2.00   sec  9.98 MBytes  83.7 Mbits/sec  919   27.3 KBytes
[  5]   2.00-3.00   sec  9.95 MBytes  83.5 Mbits/sec  954   5.71 KBytes
[  5]   3.00-4.00   sec  9.97 MBytes  83.7 Mbits/sec  939   28.7 KBytes
[  5]   4.00-5.00   sec  9.97 MBytes  83.6 Mbits/sec  951   17.3 KBytes
[  5]   5.00-6.00   sec  9.99 MBytes  83.8 Mbits/sec  913   31.5 KBytes
[  5]   6.00-7.00   sec  9.96 MBytes  83.5 Mbits/sec  956   20.1 KBytes
[  5]   7.00-8.00   sec  10.0 MBytes  83.9 Mbits/sec  913   33.0 KBytes
[  5]   8.00-9.00   sec  9.97 MBytes  83.6 Mbits/sec  945   24.4 KBytes
[  5]   9.00-10.00  sec  9.99 MBytes  83.8 Mbits/sec  916   34.4 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval   Transfer Bitrate Retr
[  5]   0.00-10.00  sec  98.7 MBytes  82.8 Mbits/sec  9337 sender
[  5]   0.00-10.25  sec  98.7 MBytes  80.8 Mbits/sec  receiver

iperf Done.
gold,pts,/home/jmg,503$iperf3 -R -c 192.168.0.80
Connecting to host 192.168.0.80, port 5201
Reverse mode, remote host 192.168.0.80 is sending
[  5] local 192.168.0.2 port 51024 connected to 192.168.0.80 port 5201
[ ID] Interval   Transfer Bitrate
[  5]   0.00-1.00   sec  9.69 MBytes  81.3 Mbits/sec
[  5]   1.00-2.00   sec  10.7 MBytes  89.8 Mbits/sec
[  5]   2.00-3.00   sec  10.7 MBytes  89.8 Mbits/sec
[  5]   3.00-4.00   sec  10.7 MBytes  89.8 Mbits/sec
[  5]   4.00-5.00   sec  10.7 MBytes  89.8 Mbits/sec
[  5]   5.00-6.00   sec  10.7 MBytes  89.8 Mbits/sec
[  5]   6.00-7.00   sec  10.7 MBytes  89.8 Mbits/sec
[  5]   7.00-8.00   sec  10.4 MBytes  87.6 Mbits/sec
[  5]   8.00-9.00   sec  10.7 MBytes  89.9 Mbits/sec
[  5]   9.00-10.00  sec  10.7 MBytes  89.8 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval   Transfer Bitrate Retr
[  5]   0.00-10.00  sec   106 MBytes  88.9 Mbits/sec  1381 sender
[  5]   0.00-10.00  sec   106 MBytes  88.7 Mbits/sec  receiver

iperf Done.

As you can see, it matches what I measured earlier.

And just to prove that the machine CAN move 100MB/sec, I've run iperf3
using the onboard wired ethernet...  I need multiple interfaces, which is
why I'm bothering trying to get USB3 ethernet working.

This is using the onboard bge interface.  It's IP is .79:
gold,pts,/home/jmg,507$iperf3 -c 192.168.0.79
Connecting to host 192.168.0.79, port 5201
[  5] local 192.168.0.2 port 61500 connected to 192.168.0.79 port 5201
[ ID] Interval   Transfer Bitrate Retr  Cwnd
[  5]   0.00-1.00   sec   101 MBytes   850 Mbits/sec0488 KBytes
[  5]   1.00-2.00   sec   112 MBytes   940 Mbits/se

Re: Is there any error checking on swap?

2020-07-12 Thread John-Mark Gurney
Xin Li wrote this message on Sun, Jul 12, 2020 at 15:06 -0700:
> 
> 
> On 7/12/20 12:29 AM, John-Mark Gurney wrote:
> > bob prohaska wrote this message on Sat, Jul 11, 2020 at 20:33 -0700:
> >> Is there any error checking on swap traffic, along the lines of
> >> a checksum or parity test? 
> >>
> >> Just curious what happens if a page written out is corrupted  when
> >> it comes back.
> > 
> > Looks like it doesn't:
> > https://svnweb.freebsd.org/base/head/sys/vm/swap_pager.c?annotate=361965#l1389
> 
> Technically one can enable checks with e.g. geli(8), but note that the
> geli(8) provider will not "prime" the HMAC data so attaching the device
> will immediately spam the system log with some authentication errors due
> to GEOM tasting (because the kernel would try to read locations that
> potentially contain metadata for other GEOM providers).

Yeah, and enabling auth w/ geli(8) is problematic, as it creates a
disconnect between layers, as it expands 4096 byte writes to 9 512
byte sector writes, if you're using a 4k drive (which is pretty much
the only thing sold these days), the performance is likely to be
pretty terrible...

see:
https://svnweb.freebsd.org/base/head/sys/geom/eli/g_eli_integrity.c?annotate=361481#l56

So, it'd be doable, but less than ideal...

> For the case of swap, since the write is always page sized, it's
> probably optimal to implement something in the swap layer itself and
> store the expected checksums in memory.

+1

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."


signature.asc
Description: PGP signature


Re: slow USB 3.0 on -current

2020-07-12 Thread John-Mark Gurney
Kevin Oberman wrote this message on Sun, Jul 12, 2020 at 16:24 -0700:
> On Sun, Jul 12, 2020 at 2:55 PM John-Mark Gurney  wrote:
> 
> > Hans Petter Selasky wrote this message on Sun, Jul 12, 2020 at 09:57 +0200:
> > > On 2020-07-12 00:44, John-Mark Gurney wrote:
> > > > Hello,
> > > >
> > > > I'm having issues getting good ethernet performance from a USB ethernet
> > > > adapter (ure) under FreeBSD on an HP EliteDesk 705 G2 Mini[1].  It's an
> > > > AMD PRO A10-8700B based system using the AMD A78 FCH chipset.
> > > >
> > > > Under FreeBSD -current (r362596), 12.1-R and 11.4-R, the RealTek USB
> > > > adapter only gets around 10MB/sec performance.  During the transfer,
> > > > the CPU usage is only around 3-5%, so it's definitely not CPU bound.
> > > >
> > > > I have tested Windows 10 and NetBSD 9.0 performance, and both provide
> > > > 100MB/sec+ w/o troubles.
> > > >
> > > > I have attached dmesg from both FreeBSD -current and NetBSD 9.0.
> > > >
> > > > Any hints on how to fix this?
> > > >
> > > > This may be related, but I'm also having issues w/ booting when I have
> > > > both a SD USB 2.0 card reader AND the ure plugged into USB 3.0 ports.
> > > >
> > > > If I move the SD card reader to USB 2.0, the umass device will attach
> > > > and work.  I have also attached a clip of the dmesg from that
> > > > happening.
> > > >
> > > > Has anyone else seen this issue?  Ideas or thoughts on how to resolve
> > > > the performance issues?
> > >
> > > Can you check the output from ifconfig. What is the actual link speed. I
> > > suspect it has something to do with the MII bus code/implementation.
> >
> > ifconfig is reporting it's 1000baseT.
> >
> > > Also check output from "vmstat -i" during usage to see if the number of
> > > IRQ/s is low.
> >
> > Not sure what is considered low, but I'm seeing consistently around
> > 7800 int/s for xhci0.
> >
> This is just for clarification, but is 'MB' MBytes? In the networking world
> that is what it would mean, but the context leads me to think that you mean
> Mbits. It's also possible that some numbers are in bits and some in Bytes,
> causing real confusion. I'm sure that 1000baseT is bits, of course.

MB means megabytes.. I would use Mbps for bits...  so, on Win10 and
NetBSD, I'm able to get 100 MBytes/sec on Win10/NetBSD, and FreeBSD,
I'm only getting a tenth the capability of gige at 9-10 MBytes/sec...

I'll note that fetch reports numbers of MBps, which is one of the tools
I've been using for testing.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: slow USB 3.0 on -current

2020-07-12 Thread John-Mark Gurney
Hans Petter Selasky wrote this message on Sun, Jul 12, 2020 at 09:57 +0200:
> On 2020-07-12 00:44, John-Mark Gurney wrote:
> > Hello,
> > 
> > I'm having issues getting good ethernet performance from a USB ethernet
> > adapter (ure) under FreeBSD on an HP EliteDesk 705 G2 Mini[1].  It's an
> > AMD PRO A10-8700B based system using the AMD A78 FCH chipset.
> > 
> > Under FreeBSD -current (r362596), 12.1-R and 11.4-R, the RealTek USB
> > adapter only gets around 10MB/sec performance.  During the transfer,
> > the CPU usage is only around 3-5%, so it's definitely not CPU bound.
> > 
> > I have tested Windows 10 and NetBSD 9.0 performance, and both provide
> > 100MB/sec+ w/o troubles.
> > 
> > I have attached dmesg from both FreeBSD -current and NetBSD 9.0.
> > 
> > Any hints on how to fix this?
> > 
> > This may be related, but I'm also having issues w/ booting when I have
> > both a SD USB 2.0 card reader AND the ure plugged into USB 3.0 ports.
> > 
> > If I move the SD card reader to USB 2.0, the umass device will attach
> > and work.  I have also attached a clip of the dmesg from that
> > happening.
> > 
> > Has anyone else seen this issue?  Ideas or thoughts on how to resolve
> > the performance issues?
> 
> Can you check the output from ifconfig. What is the actual link speed. I 
> suspect it has something to do with the MII bus code/implementation.

ifconfig is reporting it's 1000baseT.

> Also check output from "vmstat -i" during usage to see if the number of 
> IRQ/s is low.

Not sure what is considered low, but I'm seeing consistently around
7800 int/s for xhci0.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Is there any error checking on swap?

2020-07-12 Thread John-Mark Gurney
bob prohaska wrote this message on Sun, Jul 12, 2020 at 08:37 -0700:
> On Sun, Jul 12, 2020 at 12:29:12AM -0700, John-Mark Gurney wrote:
> > bob prohaska wrote this message on Sat, Jul 11, 2020 at 20:33 -0700:
> > > Is there any error checking on swap traffic, along the lines of
> > > a checksum or parity test? 
> > > 
> > > Just curious what happens if a page written out is corrupted  when
> > > it comes back.
> > 
> > Looks like it doesn't:
> > https://svnweb.freebsd.org/base/head/sys/vm/swap_pager.c?annotate=361965#l1389
> > 
> 
> Certainly nothing about parity or checksums in the comments.
> All faith in the hardware, I guess
> 
> Thanks for writing!

It probably wouldn't be too hard to add the feature...  Just expand the
page entry to store a fletcher checksum or the like...

Another option would be to try to use swap on ZFS, and use ZFS's builtin
checksum feature:
https://wiki.freebsd.org/RootOnZFS#ZFS_Swap_Volume

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Is there any error checking on swap?

2020-07-12 Thread John-Mark Gurney
bob prohaska wrote this message on Sat, Jul 11, 2020 at 20:33 -0700:
> Is there any error checking on swap traffic, along the lines of
> a checksum or parity test? 
> 
> Just curious what happens if a page written out is corrupted  when
> it comes back.

Looks like it doesn't:
https://svnweb.freebsd.org/base/head/sys/vm/swap_pager.c?annotate=361965#l1389

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


slow USB 3.0 on -current

2020-07-11 Thread John-Mark Gurney
Hello,

I'm having issues getting good ethernet performance from a USB ethernet
adapter (ure) under FreeBSD on an HP EliteDesk 705 G2 Mini[1].  It's an
AMD PRO A10-8700B based system using the AMD A78 FCH chipset.

Under FreeBSD -current (r362596), 12.1-R and 11.4-R, the RealTek USB
adapter only gets around 10MB/sec performance.  During the transfer,
the CPU usage is only around 3-5%, so it's definitely not CPU bound.

I have tested Windows 10 and NetBSD 9.0 performance, and both provide
100MB/sec+ w/o troubles.

I have attached dmesg from both FreeBSD -current and NetBSD 9.0.

Any hints on how to fix this?

This may be related, but I'm also having issues w/ booting when I have
both a SD USB 2.0 card reader AND the ure plugged into USB 3.0 ports.

If I move the SD card reader to USB 2.0, the umass device will attach
and work.  I have also attached a clip of the dmesg from that
happening.

Has anyone else seen this issue?  Ideas or thoughts on how to resolve
the performance issues?

Thanks.

[1] https://support.hp.com/us-en/document/c04834953

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
---<>---
Copyright (c) 1992-2020 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.0-CURRENT #0 r362596: Thu Jun 25 05:02:51 UTC 2020
r...@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
FreeBSD clang version 10.0.1 (g...@github.com:llvm/llvm-project.git 
llvmorg-10.0.0-97-g6f71678ecd2)
WARNING: WITNESS option enabled, expect reduced performance.
VT(efifb): resolution 1024x768
CPU: AMD PRO A10-8700B R6, 10 Compute Cores 4C+6G(1796.67-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x660f01  Family=0x15  Model=0x60  Stepping=1
  
Features=0x178bfbff
  
Features2=0x7ed8320b
  AMD Features=0x2e500800
  AMD 
Features2=0x2febbfff,DBE,PTSC,MWAITX>
  Structured Extended Features=0x1a9
  XSAVE Features=0x1
  AMD Extended Feature Extensions ID EBX=0x1000
  SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768
  TSC: P-state invariant, performance statistics
real memory  = 8589934592 (8192 MB)
avail memory = 8034414592 (7662 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: 
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
arc4random: WARNING: initial seeding bypassed the cryptographic random device 
because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.
ioapic0: MADT APIC ID 4 != hw id 0
ioapic1: MADT APIC ID 5 != hw id 0
ioapic0  irqs 0-23
ioapic1  irqs 24-55
Launching APs: 2 3 1
Timecounter "TSC" frequency 179782 Hz quality 1000
random: entropy device external interface
WARNING: Device "kbd" is Giant locked and may be deleted before FreeBSD 13.0.
kbd1 at kbdmux0
000.44 [4342] netmap_init   netmap: loaded module
[ath_hal] loaded
nexus0
efirtc0: 
efirtc0: registered as a time-of-day clock, resolution 1.00s
cryptosoft0: 
acpi0: 
acpi0: Power Button (fixed)
cpu0:  on acpi0
atrtc0:  port 0x70-0x71 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.00s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0:  port 0x40-0x43 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0:  iomem 0xfed0-0xfed003ff irq 0,8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
acpi_ec0:  port 0x62,0x66 on acpi0
acpi_button0:  on acpi0
pcib0:  port 0xcf8-0xcff on acpi0
pci0:  on pcib0
pci0:  at device 0.2 (no driver attached)
vgapci0:  port 0x2000-0x20ff mem 
0xc000-0xc7ff,0xc800-0xc87f,0xc8b0-0xc8b3 irq 43 at 
device 1.0 on pci0
vgapci0: Boot video device
hdac0:  mem 0xc8b6-0xc8b63fff at device 1.1 on 
pci0
pcib1:  at device 2.3 on pci0
pci1:  on pcib1
bge0:  mem 
0xc882-0xc882,0xc881-0xc881,0xc880-0xc880 at device 0.0 
on pci1
bge0: CHIP ID 0x05762100; ASIC REV 0x5762; CHIP REV 0x57621; PCI-E
miibus0:  on bge0
brgphy0:  PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge0: Using defaults for TSO: 65518/35/2048
bge0: Ethernet address: fc:3f:db:07:1c:f2
pci0:  at device 8.0 (no driver attached)
hdac1:  mem 0xc8b64000-0xc8b67fff at device 9.2 on 
pci0
xhci0:  mem 0xc8b68000-0xc8b69fff at device 
16.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
xhci0: Una

Re: TLS certificates for NFS-over-TLS floating client

2020-03-28 Thread John-Mark Gurney
Rick Macklem wrote this message on Thu, Mar 26, 2020 at 14:33 +:
> John-Mark Gurney wrote:
> [lots of stuff snipped]
> >Rick Macklem wrote:
> >> I had originally planned on some "secret" in the certificate (like a CN 
> >> name
> >> that satisfies some regular expression or ???) but others convinced me that
> >> that wouldn't provide anything beyond knowing that the certificate was
> >> signed by the appropriate CA, so I have not implemented anything.
> >
> >Yeah, having a "secret" in the CN doesn't make sense, but what does make
> >sense is allowing the exports line to specify what the CN should contain
> >to be authenticated...
> >
> >Say as a corp, you issue personal certificates to everyone.  This is
> >because you require everyone to sign and/or encrypt their email w/
> >S/MIME.  Each cert includes the email address of that person, so you
> >could simply do something like:
> >/home/alice -tlscert -tlsroot /etc/company.pem -email al...@example.com
> >
> >And anyone who has the certificate w/ al...@example.com that was signed
> >by the public key in /etc/company.pem would be granted access to the
> >export /home/alice.
> >
> >If it allowed ANY cert signed by the CA specified, then that introduces
> >an authentication problem, as now if Malory is a coworker of Alice
> >could also access Alice's home directory...
> >
> >IMO, this is one auth feature that MUST be supported...
> The draft does not address user authentication, only machine authentication.
> --> ie. The certificate is used to decide if a system can do a mount.
>   Users are still identified via user credentials in the RPC message 
> header.
>   For AUTH_SYS, that still means .
>   Otherwise, you need to use Kerberos (sec=krb5[ip]).
>   You could use "tls,sec=krb5" for a mount, but the only advantage that
>   might have over "sec=krb5p" is performance, if there is hardware assist
>   for TLS or something like that.
> 
> >Now that I reread your comments, it sounds like any certificate would be
> >allowed in #2 as long as it is valid, and that would only be marginally
> >better than verification by IP, and in some ways worse, in that now any
> >user could pretend to be any other user, or you have to do something
> >crazy like have a CA per user.
> The case where I see #2 is useful is where this discussion started some weeks 
> ago.
> The example I started with was:
> /home -tls -network 192.168.1.0 -mask 255.255.255.0
> /home -tlscert
> 
> This says that machines on the local lan can mount and do not need to have
> certificates, but must use TLS so data is encrypted on the wire.
> Mounts from anywhere else (presumably laptops) are allowed so long as they 
> have a
> certificate signed by me (as in the site local CA).
> I trust these machines enough that I am willing to allow them to use AUTH_SYS,
> which is what 99.9...% of NFS mounts do now.
> (So, I'd agree that the site local certificate is not a lot better than IP 
> address
>  for client machine identity, just that it is an alternative that can be 
> useful.
>  Without TLS, a line like "/home" allows anyone to mount /home from anywhere
>  and I think you'd agree that few NFS admins. will want to do that. I'm 
> assuming
>  no external firewall for this example.)
> 
> Now, your suggestion of identifying a user via the CN and then having the
> server do all RPCs for the mount as that user is an interesting one.
> My concern w.r.t. implementing it would be interoperability.
> Put another way, if other servers such as Linux, Netapp,... don't adopt it
> (and they won't until there is a draft/RFC specifying it), it would be
> FreeBSD server specific and I'd like to avoid that.
> There was some discussion w.r.t. user authentication via certificates
> during development of the draft, but they decided to defer that work for
> now, so they could get something in place for machine authentication first.
> (If I understood the discussion on nf...@ietf.org.)

There's a fine line between machine and user authentication when you can
add -mapall=someuser.  :)

Yes, a certificate may be used to identify a machine, but if any machine
can be identified by any cert as it appears that #2 is proposed to do, then
you're not actually identifying a machine, you're identifying group of
machines...  Though per the draft, this is explicitly allowed.

It'd be nice to be able to identify particular machines though, and be
able to set the options based upon that..

> >I'm wonder if your use of the term secret was the problem, and not the
> >idea...  Anything that goes in the client cert is by defini

Re: TLS certificates for NFS-over-TLS floating client

2020-03-25 Thread John-Mark Gurney
Rick Macklem wrote this message on Wed, Mar 25, 2020 at 23:50 +:
> John-Mark Gurney wrote:
> >Rick Macklem wrote this message on Mon, Mar 23, 2020 at 23:53 +:
> >> Alexander Leidinger wrote:
> >> John-Mark Gurney  wrote:
> >> >>Rick Macklem wrote:
> >> >>> to be the best solution. The server can verify that the certificate
> >> >>> was issued by
> >> >>> the local CA. Unfortunately, if the client is compromised and the
> >> >>> certificate is copied
> >> >>> to another client, that client would gain access.
> >> >>
> >> >> This is why CRLs/OSCP is necessary, but there isn't anything you can do
> >> >> to prevent that.  This is both a better situation than what we have
> >> >> today (no auth/confidentiality), and if you tie the cert to an IP, it's
> >> >> of limited use, and better than today...
> >> >
> >> >There are multiple ways to handle that:
> >> >  - First of all, you can just validate based upon "cert is signed by
> >> >trusted CA".
> >> >  - Then you can require that the CN matches the hostname and the
> >> >reverse lookup matches.
> >> >  - Or (similar to browsers today) you can ignore the CN and require
> >> >that the hostnames of the client matches one of the subject
> >> >alternative name (SAN) entries (requires reverse DNS lookup to work
> >> >and match).
> >> At this point, I have three variants in the code (strictest to less 
> >> strict):
> >> 1 - A "-h" command line option on the server handshake daemon (called 
> >> rpctlssd).
> >>  This requires that all clients have
> >>  certificates that validate and have the FQDN acquired via reverse DNS 
> >> of
> >>  the IP address of the client for the TCP connection 
> >> (getnameinfo(NI_NAMEREQD))
> >>  in either the subjectAltName or CommonName. (I call X509_check_host()
> >>  and this is what I understand it checks.)
> >>  --> This case implies that the NFS server admin. does not "trust" the
> >> client's IP address enough to apply exports(5) line(s)to it to 
> >> decide to
> >> allow the client to do an NFS mount.
> >>  (An NFS server *might* be willing to allow client(s) to mount via any 
> >> IP address
> >>   for the #2 case below and not use this option.)
> >> 2 - Without "-h" the rpctlssd daemon passes flags into the kernel to 
> >> indicate
> >>  if the client provided a certificate and whether or not it verified.
> >>  Then the "-tlscert" option on the appropriate exports(5) line(s)
> >>  indicates that the client must have provided a certificate that 
> >> verified.
> >>  --> For this case (and #3), the server admin. is willing to "trust" 
> >> the client's
> >> IP address enough to apply exports(5) rules to it.
> >>  --> This is the case where a floating (no fixed IP) laptop could have 
> >> a
> >>certificate signed by a site local CA.
> >> 3 - Similar to #2, but uses the "-tls" option on the exports(5) line(s).
> >>  In this case, the client must use TLS so that data is encrypted on 
> >> the wire,
> >>  but does not need to have a certificate.
> >>  --> The NFS server admin. "trusts" the client IP address enough that 
> >> they
> >>are willing to allow the client to mount based on that IP, but 
> >> wants the
> >>data to be encrypted on the wire.
> >>- Avoids the bother of generating certificates for the 
> >> client(s).
> >>
> >> A part of this (as discussed in the IETF draft) is to make this easy 
> >> enough to
> >> use that it does get used. (sec=krb5p achieves both user authentication and
> >> data encryption on the wire, but does not get widely used, due to the need
> >> to run a KDC, etc).
> >>
> >> Comments on the above options is welcome, since this does need to be
> >> reviewed by users.
> >
> >Maybe I'm missing the option where the cert needs to be authenticated,
> >but matching against IP/dns name does not need to be done.  Or is this
> >a case of #2.  I'm just confused by the first point of #2 in that the
> >server admin is wiling to trust the IP address...
> Yes, that is #2. 

Re: TLS certificates for NFS-over-TLS floating client

2020-03-25 Thread John-Mark Gurney
Rick Macklem wrote this message on Mon, Mar 23, 2020 at 23:53 +:
> Alexander Leidinger wrote:
> John-Mark Gurney  wrote:
> >>Rick Macklem wrote:
> >>> to be the best solution. The server can verify that the certificate  
> >>> was issued by
> >>> the local CA. Unfortunately, if the client is compromised and the  
> >>> certificate is copied
> >>> to another client, that client would gain access.
> >>
> >> This is why CRLs/OSCP is necessary, but there isn't anything you can do
> >> to prevent that.  This is both a better situation than what we have
> >> today (no auth/confidentiality), and if you tie the cert to an IP, it's
> >> of limited use, and better than today...
> >
> >There are multiple ways to handle that:
> >  - First of all, you can just validate based upon "cert is signed by  
> >trusted CA".
> >  - Then you can require that the CN matches the hostname and the  
> >reverse lookup matches.
> >  - Or (similar to browsers today) you can ignore the CN and require  
> >that the hostnames of the client matches one of the subject  
> >alternative name (SAN) entries (requires reverse DNS lookup to work  
> >and match).
> At this point, I have three variants in the code (strictest to less strict):
> 1 - A "-h" command line option on the server handshake daemon (called 
> rpctlssd).
>  This requires that all clients have
>  certificates that validate and have the FQDN acquired via reverse DNS of
>  the IP address of the client for the TCP connection 
> (getnameinfo(NI_NAMEREQD))
>  in either the subjectAltName or CommonName. (I call X509_check_host()
>  and this is what I understand it checks.)
>  --> This case implies that the NFS server admin. does not "trust" the
> client's IP address enough to apply exports(5) line(s)to it to 
> decide to
> allow the client to do an NFS mount.
>  (An NFS server *might* be willing to allow client(s) to mount via any IP 
> address
>   for the #2 case below and not use this option.)
> 2 - Without "-h" the rpctlssd daemon passes flags into the kernel to indicate
>  if the client provided a certificate and whether or not it verified.
>  Then the "-tlscert" option on the appropriate exports(5) line(s) 
>  indicates that the client must have provided a certificate that verified.
>  --> For this case (and #3), the server admin. is willing to "trust" the 
> client's
> IP address enough to apply exports(5) rules to it.
>  --> This is the case where a floating (no fixed IP) laptop could have a
>certificate signed by a site local CA.
> 3 - Similar to #2, but uses the "-tls" option on the exports(5) line(s).
>  In this case, the client must use TLS so that data is encrypted on the 
> wire,
>  but does not need to have a certificate.
>  --> The NFS server admin. "trusts" the client IP address enough that they
>are willing to allow the client to mount based on that IP, but 
> wants the
>data to be encrypted on the wire.
>- Avoids the bother of generating certificates for the client(s).
> 
> A part of this (as discussed in the IETF draft) is to make this easy enough to
> use that it does get used. (sec=krb5p achieves both user authentication and
> data encryption on the wire, but does not get widely used, due to the need
> to run a KDC, etc).
> 
> Comments on the above options is welcome, since this does need to be
> reviewed by users.

Maybe I'm missing the option where the cert needs to be authenticated,
but matching against IP/dns name does not need to be done.  Or is this
a case of #2.  I'm just confused by the first point of #2 in that the
server admin is wiling to trust the IP address...

I'd like to see where CN or other field is freeform/provided by exports
entry, and validated to gain access to those resources...  i.e. it
doesn't matter what IP or DNS name the client is, it's based solely on
the certificate.  This would allow roaming users..  This maybe be
addressed by #2 point 2, but it isn't clear in your description that
it isn't dns tied or something else...

As the CA admin, they control what is valid/gets signed in the CN or
other fields, so this is safe to do... If you can't trust your CA to
sign only valid (to your policy) certs, then you shouldn't be using
that CA..

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: TLS certificates for NFS-over-TLS floating client

2020-03-20 Thread John-Mark Gurney
Jan Bramkamp wrote this message on Fri, Mar 20, 2020 at 18:51 +0100:
> On 20.03.20 02:44, Russell L. Carter wrote:
> > Here I commit heresy, by A) top posting, and B) by just saying, why
> > not make it easy, first, to tunnel NFSv4 sessions through
> > e.g. net/wireguard or sysutils/spiped?  NFS is point to point.
> > Security infrastructure that actually works understands the shared
> > secret model.

VPN tunneling doesn't provide the security that most people thinks it
does...  It requires complicated configuration, and often doesn't
provide e2e protections.

> Why not use IPsec in transport mode instead of a tunnel? It avoids 
> unnecessary overhead and is already implemented in the kernel. It should 
> be enough to "just" require IPsec for TCP port 2049 and run a suitable 
> key exchange daemon.

Because IPsec is a PITA to configure and work, and lots of consumer OSes
don't make it at all easy.

Also, you forget that FreeBSD has ktls, which usees the same crypto
offload engine that IPsec does, so it will effectively have similar
overhead, and might actually perform better due to TLS having a 16k
record encryption size vs IPsec limiting itself to packet size, usually
1500, though possibly 9k if you're using jumbo frames...

I fully support doing NFS over TLS.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: TLS certificates for NFS-over-TLS floating client

2020-03-20 Thread John-Mark Gurney
Rick Macklem wrote this message on Thu, Mar 19, 2020 at 23:41 +:
> John-Mark Gurney wrote:
> >Rick Macklem wrote this message on Wed, Mar 04, 2020 at 03:15 +:
> >> I am slowly trying to understand TLS certificates and am trying to figure
> >> out how to do the following:
> >> -> For an /etc/exports file with...
> >> /home -tls -network 192.168.1.0 -mask 255.255.255.0
> >> /home -tlscert
> >
> >Are you looking at implementing draft-cel-nfsv4-rpc-tls?
> Yes. The 2 week out of date (I can only do commits once in a while these 
> days) can
> be found in FreeBSD's subversion under base/projects/nfs-over-tls.

Nifty.

And looks like you know about the ktls stuff, nice...

> >> This syntax isn't implemented yet, but the thinking is that clients on the
> >> 192.168.1 subnet would use TLS, but would not require a certificate.
> >> For access from anywhere else, the client(s) would be required to have a
> >> certificate.
> >>
> >> A typical client mounting from outside of the subnet might be my laptop,
> >> which is using wifi and has no fixed IP/DNS name.
> >> --> How do you create a certificate that the laptop can use, which the NFS
> >>server can trust enough to allow the mount?
> >> My thinking is that a "secret" value can be put in the certificate that 
> >> the NFS
> >> server can check for.
> >> The simplest way would be a fairly long list of random characters in the
> >> organizationName and/or organizationUnitName field(s) of the subject name.
> >> Alternately, it could be a newly defined extension for X509v3, I think?
> >>
> >> Now, I'm not sure, but I don't think this certificate can be created via
> >> a trust authority such that it would "verify". However, the server can
> >> look for the "secret" in the certificate and allow the mount based on that.
> >>
> >> Does this sound reasonable?
> >
> >Without a problem statement or what you're trying to accomplish, it's
> >hard to say if it is.
> The problem I was/am trying to solve was a way for NFS clients without a
> fixed IP/DNS name could have a certificate to allow access to the NFS server.
> As suggested by others, having a site local CA created by the NFS admin. 
> seemed

Yes, I totally agree w/ this as the best solution.  It also allows
private hostnames to be used w/o leaking outside the org..

It'd be nice to have better tooling around the CA though.  I still
haven't found any good tools that make a CA simple to use for small
installs...  (and by simple, I mean single init command, and single
command to issue a cert or generate a key/cert pair, all of them are
like, make all thesse directories, edit these files, and run these
comlicated commands)

Another option is to just use self sign certs and manually add each
one for a host, not sure how difficult this would be in code...  This
would probably be easiest w/ a small number of clients, eliminates the
difficulty of doing a CRL/OSCP, as if a client gets compromised, you
just delete their cert till a new one is issued...

> to be the best solution. The server can verify that the certificate was 
> issued by
> the local CA. Unfortunately, if the client is compromised and the certificate 
> is copied
> to another client, that client would gain access.

This is why CRLs/OSCP is necessary, but there isn't anything you can do
to prevent that.  This is both a better situation than what we have
today (no auth/confidentiality), and if you tie the cert to an IP, it's
of limited use, and better than today...

> --> I've thought of having the client keep the certificate encrypted in a 
> file and
>require the "user" of the client type in a passphrase to unencrypt the 
> certificate
>so that it can be used by the daemon in the client that handles the 
> client side
>of the TLS handshake, but I have not implemented this.
>--> This would at least subvert the simple case of the certificate 
> file being copied
>   to a different client and being used to mount the NFS server, 
> but if the
>   client is compromised, then the passphrase could be captured 
> and...

Exactly.  Just make sure that it's clear how to handle a revoked
certificate when this happens, and you're good...

> >> Also, even if the NFS client/server have fixed IP addresses with well known
> >> DNS names, it isn't obvious to me how signed certificates can be acquired
> >> for them?
> >> (Lets Encrypt expects the Acme protocol to work and that seems to be
> >>  web site/http specific?)
> >
> >There is DNS challenges that can be

Re: TLS certificates for NFS-over-TLS floating client

2020-03-19 Thread John-Mark Gurney
Rick Macklem wrote this message on Wed, Mar 04, 2020 at 03:15 +:
> I am slowly trying to understand TLS certificates and am trying to figure
> out how to do the following:
> -> For an /etc/exports file with...
> /home -tls -network 192.168.1.0 -mask 255.255.255.0
> /home -tlscert

Are you looking at implementing draft-cel-nfsv4-rpc-tls?

> This syntax isn't implemented yet, but the thinking is that clients on the
> 192.168.1 subnet would use TLS, but would not require a certificate.
> For access from anywhere else, the client(s) would be required to have a
> certificate.
> 
> A typical client mounting from outside of the subnet might be my laptop,
> which is using wifi and has no fixed IP/DNS name.
> --> How do you create a certificate that the laptop can use, which the NFS
>server can trust enough to allow the mount?
> My thinking is that a "secret" value can be put in the certificate that the 
> NFS
> server can check for.
> The simplest way would be a fairly long list of random characters in the
> organizationName and/or organizationUnitName field(s) of the subject name.
> Alternately, it could be a newly defined extension for X509v3, I think?
> 
> Now, I'm not sure, but I don't think this certificate can be created via
> a trust authority such that it would "verify". However, the server can
> look for the "secret" in the certificate and allow the mount based on that.
> 
> Does this sound reasonable?

Without a problem statement or what you're trying to accomplish, it's
hard to say if it is.

> Also, even if the NFS client/server have fixed IP addresses with well known
> DNS names, it isn't obvious to me how signed certificates can be acquired
> for them?
> (Lets Encrypt expects the Acme protocol to work and that seems to be
>  web site/http specific?)

There is DNS challenges that can be used.  I use them to obtain certs
for SMTP and SIP servers...  using nsupdate, this is relatively easy to
automate pushing the challenges to a DNS server, and I now use DNS
challenges for everything, including https.

> Thanks for any help with this, rick

Let me know if you'd like to hop on a call about this.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: FreeBSD Friday

2019-11-22 Thread John-Mark Gurney
Robert wrote this message on Fri, Nov 22, 2019 at 22:24 +0100:
> Thanks all, in the process of installing 13-current snapshot.

Just a FYI, I've written a utility, snapaid, that makes is much easier
to fetch the latest snapshot:
https://github.com/jmgurney/snapaid

Hope people find it useful.

> On Fri, 22 Nov 2019 at 22:21, Clay Daniels 
> wrote:
> 
> > Just my two cents worth, but I think you would have more fun and learn new
> > stuff if you followed weekly the 13.0 Current snapshots all shown at the
> > same place, most every Thursday:
> >
> > https://www.freebsd.org/where.html
> >
> >  Pick the i386 for your intel laptop.  You will get the most up to date
> > install short of compiling the whole thing daily. Either way, have fun!
> >
> > Best Wishes.
> >
> > Clay
> >
> > On Fri, Nov 22, 2019 at 1:26 PM Thomas The Tank Engine <
> > rgleeson...@gmail.com> wrote:
> >
> >> Hello all,
> >>
> >> Apologies if this is going to the wrong list, but I would like to make
> >> FreeBSD my every day operating system. After researching, I became aware I
> >> need a commit that was made only 2 days ago. It is suppose to be merged to
> >> 12-STABLE soon, but I'd like to install today.
> >>
> >> The commit I need is referenced here:
> >> https://github.com/wjguo/freebsd/pull/1#issuecomment-556060040
> >>
> >> I haven't used FreeBSD since 4.4-RELEASE, and I think 12-stable would
> >> probably be a better option ... I am willing to try out 13-CURRENT though.
> >>
> >> Does anyone have advice? Could someone point me to snapshot of 13-CURRENT
> >> that'd include the commit I need?

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: poudriere, swap full and top says memory is free ?

2019-09-14 Thread John-Mark Gurney
Kurt Jaeger wrote this message on Sat, Sep 14, 2019 at 19:38 +0200:
> - a poudriere build
> - of a list of ports
> - on 12.0-RELEASE-p10
> - on a 4 core+4 hyperthreads CPU, an Intel(R) Xeon(R) CPU E3-1230 v6
>   @ 3.50GHz
> - with 32 GB RAM
> - zpool with 2x 500 GB SSDs as a mirror
> 
> and right now, this can be seen:
> 
> last pid: 90922;  load averages:  5.02,  5.14,  5.73up 0+03:53:08  
> 19:31:05
> 82 processes:  6 running, 76 sleeping
> CPU: 60.6% user,  0.0% nice,  2.1% system,  0.0% interrupt, 37.3% idle
> Mem: 4598M Active, 2854M Inact, 11G Laundry, 6409M Wired, 6375M Free
> ARC: 3850M Total, 1721M MFU, 2090M MRU, 665K Anon, 19M Header, 19M Other
>  3406M Compressed, 3942M Uncompressed, 1.16:1 Ratio
> Swap: 18G Total, 18G Used, 396K Free, 99% Inuse, 68K In
> 
> So: Swap is full, approx. 6 GB memory is reported as free.
> 
> This is surprising. Can I somehow tune this in any way, so that
> the memory available is used for the build ? Or is the problem somewhere
> else ?

Are you sure that this hasn't just recently completed a large link of
something like Chromium?  There are known to be compiles that can take
many GB's of memory and if they recently exited, there hasn't been time
to swap stuff back in...  or is this the steady state over the entire
compile?

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: maxswzone NOT used correctly and defaults incorrect?

2018-11-24 Thread John-Mark Gurney
Konstantin Belousov wrote this message on Sat, Nov 24, 2018 at 12:40 +0200:
> On Sat, Nov 24, 2018 at 01:04:29AM -0800, John-Mark Gurney wrote:
> > I have an BeagleBoard Black.  I'm running a recent snapshot:
> > FreeBSD generic 13.0-CURRENT FreeBSD 13.0-CURRENT r340239 GENERIC  arm
> > 
> > aka:
> > FreeBSD-13.0-CURRENT-arm-armv7-BEAGLEBONE-20181107-r340239.img.xz
> > 
> > It has 512MB of memory on board.  I created a 4GB swap file.  According
> > to loader(8), this should be the default capable:
> >in bytes of KVA space.  If no value is provided, the 
> > system
> >allocates enough memory to handle an amount of swap that
> >corresponds to eight times the amount of physical memory
> >present in the system.
> > 
> > avail memory = 505909248 (482 MB)
> > 
> > but I get this:
> > warning: total configured swap (1048576 pages) exceeds maximum recommended 
> > amount (248160 pages).
> > warning: increase kern.maxswzone or reduce amount of swap.
> > 
> > So, this appears that it's only 2x amount of memory, NOT 8x like the
> > documentation says.
> > 
> > When running make in sbin/ggate/ggated, make consumes a large amount
> > of memory.  Before the OOM killer just kicked in, top showed:
> > Mem: 224M Active, 4096 Inact, 141M Laundry, 121M Wired, 57M Buf, 2688K Free
> > Swap: 1939M Total, 249M Used, 1689M Free, 12% Inuse, 1196K Out
> > 
> >   PIDUID  THR PRI NICE   SIZERES STATETIMEWCPU COMMAND
> >  1029   10011  440   594M  3848K RUN  2:03  38.12% make
> > 
> > swapinfo -k showed:
> > /dev/md99 4194304   254392  3939912 6%
> > 
> > sysctl:
> > vm.swzone: 4466880
> > vm.swap_maxpages: 496320
> > kern.maxswzone: 0
> > 
> > dmesg when OOM strikes:
> > swap blk zone exhausted, increase kern.maxswzone
> > pid 1029 (make), uid 1001, was killed: out of swap space
> > pid 984 (bash), uid 1001, was killed: out of swap space
> > pid 956 (bash), uid 1001, was killed: out of swap space
> > pid 952 (sshd), uid 0, was killed: out of swap space
> > pid 1043 (bash), uid 1001, was killed: out of swap space
> > pid 626 (dhclient), uid 65, was killed: out of swap space
> > pid 955 (sshd), uid 1001, was killed: out of swap space
> > pid 1025 (bash), uid 1001, was killed: out of swap space
> > swblk zone ok
> > lock order reversal:
> >  1st 0xd374d028 filedesc structure (filedesc structure) @ 
> > /usr/src/sys/kern/sys_generic.c:1451
> >  2nd 0xd41a5bc4 devfs (devfs) @ /usr/src/sys/kern/vfs_vnops.c:1513
> > stack backtrace:
> > swap blk zone exhausted, increase kern.maxswzone
> > pid 981 (tmux), uid 1001, was killed: out of swap space
> > pid 983 (tmux), uid 1001, was killed: out of swap space
> > pid 1031 (bash), uid 1001, was killed: out of swap space
> > pid 580 (dhclient), uid 0, was killed: out of swap space
> > swblk zone ok
> > swap blk zone exhausted, increase kern.maxswzone
> > pid 577 (dhclient), uid 0, was killed: out of swap space
> > pid 627 (devd), uid 0, was killed: out of swap space
> > swblk zone ok
> > swap blk zone exhausted, increase kern.maxswzone
> > pid 942 (getty), uid 0, was killed: out of swap space
> > swblk zone ok
> > swap blk zone exhausted, increase kern.maxswzone
> > pid 1205 (init), uid 0, was killed: out of swap space
> > swblk zone ok
> > swap blk zone exhausted, increase kern.maxswzone
> > pid 1206 (init), uid 0, was killed: out of swap space
> > swblk zone ok
> > swap blk zone exhausted, increase kern.maxswzone
> > swblk zone ok
> > swap blk zone exhausted, increase kern.maxswzone
> > swblk zone ok
> > 
> > So, as you can see, despite having plenty of swap, and swap usage being
> > well below any of the maximums, the OOM killer kicked in, and killed off
> > a bunch of processes.
> OOM is guided by the pagedaemon progress, not by the swap amount left.
> If the system cannot meet the pagedaemon targetp by doing
> $(sysctl vm.pageout_oom_seq) back-to-back page daemon passes,
> it declares OOM condition. E.g. if you have very active process which
> keeps a lot of active memory by referencing the pages, and simultenously
> a slow or stuck swap device, then you get into this state.
> 
> Just by looking at the top stats, you have a single page in the inactive
> queue, which means that pagedaemon desperately frees clean pages and
> moves dirty pages into the laundry.  Also, you have relatively large
> laundry queue, which supports the theory a

maxswzone NOT used correctly and defaults incorrect?

2018-11-24 Thread John-Mark Gurney
I have an BeagleBoard Black.  I'm running a recent snapshot:
FreeBSD generic 13.0-CURRENT FreeBSD 13.0-CURRENT r340239 GENERIC  arm

aka:
FreeBSD-13.0-CURRENT-arm-armv7-BEAGLEBONE-20181107-r340239.img.xz

It has 512MB of memory on board.  I created a 4GB swap file.  According
to loader(8), this should be the default capable:
   in bytes of KVA space.  If no value is provided, the system
   allocates enough memory to handle an amount of swap that
   corresponds to eight times the amount of physical memory
   present in the system.

avail memory = 505909248 (482 MB)

but I get this:
warning: total configured swap (1048576 pages) exceeds maximum recommended 
amount (248160 pages).
warning: increase kern.maxswzone or reduce amount of swap.

So, this appears that it's only 2x amount of memory, NOT 8x like the
documentation says.

When running make in sbin/ggate/ggated, make consumes a large amount
of memory.  Before the OOM killer just kicked in, top showed:
Mem: 224M Active, 4096 Inact, 141M Laundry, 121M Wired, 57M Buf, 2688K Free
Swap: 1939M Total, 249M Used, 1689M Free, 12% Inuse, 1196K Out

  PIDUID  THR PRI NICE   SIZERES STATETIMEWCPU COMMAND
 1029   10011  440   594M  3848K RUN  2:03  38.12% make

swapinfo -k showed:
/dev/md99 4194304   254392  3939912 6%

sysctl:
vm.swzone: 4466880
vm.swap_maxpages: 496320
kern.maxswzone: 0

dmesg when OOM strikes:
swap blk zone exhausted, increase kern.maxswzone
pid 1029 (make), uid 1001, was killed: out of swap space
pid 984 (bash), uid 1001, was killed: out of swap space
pid 956 (bash), uid 1001, was killed: out of swap space
pid 952 (sshd), uid 0, was killed: out of swap space
pid 1043 (bash), uid 1001, was killed: out of swap space
pid 626 (dhclient), uid 65, was killed: out of swap space
pid 955 (sshd), uid 1001, was killed: out of swap space
pid 1025 (bash), uid 1001, was killed: out of swap space
swblk zone ok
lock order reversal:
 1st 0xd374d028 filedesc structure (filedesc structure) @ 
/usr/src/sys/kern/sys_generic.c:1451
 2nd 0xd41a5bc4 devfs (devfs) @ /usr/src/sys/kern/vfs_vnops.c:1513
stack backtrace:
swap blk zone exhausted, increase kern.maxswzone
pid 981 (tmux), uid 1001, was killed: out of swap space
pid 983 (tmux), uid 1001, was killed: out of swap space
pid 1031 (bash), uid 1001, was killed: out of swap space
pid 580 (dhclient), uid 0, was killed: out of swap space
swblk zone ok
swap blk zone exhausted, increase kern.maxswzone
pid 577 (dhclient), uid 0, was killed: out of swap space
pid 627 (devd), uid 0, was killed: out of swap space
swblk zone ok
swap blk zone exhausted, increase kern.maxswzone
pid 942 (getty), uid 0, was killed: out of swap space
swblk zone ok
swap blk zone exhausted, increase kern.maxswzone
pid 1205 (init), uid 0, was killed: out of swap space
swblk zone ok
swap blk zone exhausted, increase kern.maxswzone
pid 1206 (init), uid 0, was killed: out of swap space
swblk zone ok
swap blk zone exhausted, increase kern.maxswzone
swblk zone ok
swap blk zone exhausted, increase kern.maxswzone
swblk zone ok

So, as you can see, despite having plenty of swap, and swap usage being
well below any of the maximums, the OOM killer kicked in, and killed off
a bunch of processes.

It also looks like the algorithm for calculating kern.maxswzone is not
correct.

I just tried to run the system w/:
kern.maxswzone: 21474836

and it again died w/ plenty of swap free:
/dev/md99 4194304   238148  3956156 6%

This time I had vmstat -z | grep sw running, and saw:
swpctrie:48,  62084, 145, 270, 203,   0,   0
swblk:   72,  62040,   56357,  18,   56587,   0,   0

after the system died, I logged back in as see:
swpctrie:48,  62084,  28, 387, 240,   0,   0
swblk:   72,  62040, 175,   61865,   62957,  16,   0

so, it clearly ran out of swblk space VERY early, when only consuming
around 232MB of swap...

Hmm... it looks like swblk and swpctrie are not affected by the setting
of kern.maxswzone...  I just set it to:
kern.maxswzone: 85899344

and the limits for the zones did not increase at ALL:
swpctrie:48,  62084,   0,   0,   0,   0,   0
swblk:   72,  62040,   0,   0,   0,   0,   0

Thoughts?

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: UFS panics

2018-10-27 Thread John-Mark Gurney
Leandro wrote this message on Wed, Oct 24, 2018 at 12:22 -0300:
> I'm seeing a kernel panic when trying to move a specific file.
> 
> panic: Bad effnlink fip 0xc004a2c69a00, fdp 0xc00497093be0,
> tdp 0xc004be6295a0
> cpuid = 72
> time = 1540283798
> KDB: stack backtrace:
> 0xe000adb8dcc0: at .kdb_backtrace+0x5c
> 0xe000adb8ddf0: at .vpanic+0x1b4
> 0xe000adb8deb0: at .panic+0x38
> 0xe000adb8df40: at .ufs_readdir+0x2f24
> 0xe000adb8e1b0: at .VOP_RENAME_APV+0x190
> 0xe000adb8e240: at .kern_renameat+0x3c0
> 0xe000adb8e540: at .sys_rename+0x2c
> 0xe000adb8e5c0: at .trap+0x65c
> 0xe000adb8e780: at .powerpc_interrupt+0x290
> 0xe000adb8e820: user SC trap by 0x81010b7b8: srr1=0x9000f032
> r1=0x3fffc490 cr=0x3428 xer=0x2000
> ctr=0x81010b7b0 r2=0x8102c5950
> KDB: enter: panic
> 
> Using 'ls' or 'rm' on the file gives a "Bad file descriptor" error.
> Using 'cat', I get another panic, but now during open.
> Everything indicates that the file system is in an inconsistent state.
> 
> Therefore, I would just like to ask the following: is it expected for
> kernel panics to happen when there are errors in the file system?

Have you unmounted the file system, and tried to do a fsck on it?  That
you're getting a bad file descriptor sounds like it might be a directory
entry pointing to a bad inode.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: which way to update export_args structure?

2018-10-23 Thread John-Mark Gurney
Brooks Davis wrote this message on Mon, Oct 22, 2018 at 16:05 +:
> > +   switch (len) {
> > +   case (sizeof(struct oexport_args)):
> > +   case (sizeof(struct o2export_args)):
> > +   memset(, 0, sizeof(export));
> 
> I think this is now redundant.
> 
> > +   memset(, 0, sizeof(o2export));
> 
> This is certainly redundant given that you immediately copy over it.

This is not redundant if sizeof oexport_args < sizeof o2export, and
len == sizeof oexport_args...  It zeros out the remaining of the buffer..

> > +       memcpy(, bufp, len);

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."


signature.asc
Description: PGP signature


Re: Speed problems with both system openssl and security/openssl-devel

2018-09-22 Thread John-Mark Gurney
Lev Serebryakov wrote this message on Mon, Sep 17, 2018 at 23:52 +0300:
> Thursday, September 13, 2018, 2:46:46 AM, you wrote:
> 
> >   Linux have openssl 1.1.0f, and  I've tried both system /usr/bin/openssl 
> > (1.0.2p)
> > and /usr/local/bin/openssl from security/openssl-devel port (1.1.0i), 
> > results are
> > virtually the same. I have "ASM" and "SSE2" options enabled in port.
> 
> >  What happens here? Why does FreeBSD's build of openssl use AES-NI so
> > inefficient?
>  More datapoints.
> 
> (1) aes-256-cbc behaves really wired. Time output is
> completely bogus without "-elapsed" and speed is unbelievably low with
> "-elapsed". aes-256-gcm doesn't have this anomaly

This is because you're likely using /dev/crypto for the operations instead
of software...

$openssl engine 
(cryptodev) BSD cryptodev engine
(dynamic) Dynamic engine loading support

The times below are mesured on how much cpu time openssl spent, while
all the work was done in the kernel...

if you disable cryptodev usage, you should see better performance...

> without "-elapsed" (please note "in 0.xxs" here):
> 
> Doing aes-256-cbc for 3s on 16 size blocks: 503555 aes-256-cbc's in 0.60s
> Doing aes-256-cbc for 3s on 64 size blocks: 520386 aes-256-cbc's in 0.54s
> Doing aes-256-cbc for 3s on 256 size blocks: 435106 aes-256-cbc's in 0.44s
> Doing aes-256-cbc for 3s on 1024 size blocks: 242832 aes-256-cbc's in 0.38s
> Doing aes-256-cbc for 3s on 8192 size blocks: 49087 aes-256-cbc's in 0.09s
> ...
> aes-256-cbc  13393.26k61782.64k   254599.17k   663093.25k  4289287.51k
> 
> Doing aes-256-gcm for 3s on 16 size blocks: 12051311 aes-256-gcm's in 3.03s
> Doing aes-256-gcm for 3s on 64 size blocks: 6428598 aes-256-gcm's in 3.04s
> Doing aes-256-gcm for 3s on 256 size blocks: 2122316 aes-256-gcm's in 3.00s
> Doing aes-256-gcm for 3s on 1024 size blocks: 610443 aes-256-gcm's in 3.13s
> Doing aes-256-gcm for 3s on 8192 size blocks: 75836 aes-256-gcm's in 3.03s
> ...
> aes-256-gcm  63611.04k   135380.66k   181104.30k   199531.13k   204947.96k
> 
> with "-elapsed":
> 
> Doing aes-256-cbc for 3s on 16 size blocks: 493829 aes-256-cbc's in 3.01s
> Doing aes-256-cbc for 3s on 64 size blocks: 530550 aes-256-cbc's in 3.06s
> Doing aes-256-cbc for 3s on 256 size blocks: 426699 aes-256-cbc's in 3.01s
> Doing aes-256-cbc for 3s on 1024 size blocks: 243305 aes-256-cbc's in 3.03s
> Doing aes-256-cbc for 3s on 8192 size blocks: 48069 aes-256-cbc's in 3.01s
> ...
> aes-256-cbc   2626.91k11087.41k36317.07k82191.94k   130919.48k
> 
> Doing aes-256-gcm for 3s on 16 size blocks: 12041385 aes-256-gcm's in 3.08s
> Doing aes-256-gcm for 3s on 64 size blocks: 6445757 aes-256-gcm's in 3.05s
> Doing aes-256-gcm for 3s on 256 size blocks: 2129499 aes-256-gcm's in 3.01s
> Doing aes-256-gcm for 3s on 1024 size blocks: 587396 aes-256-gcm's in 3.01s
> Doing aes-256-gcm for 3s on 8192 size blocks: 75806 aes-256-gcm's in 3.03s
> ...
> aes-256-gcm  62590.75k   135047.68k   181245.26k   199977.06k   204866.89k

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


buffer device_printf...

2018-08-16 Thread John-Mark Gurney
On my arm64 board, various console messages get intermixed because of
SMP.  This makes messages harder to read than the could be.  One issue
is that device_printf uses two printf's, one for the device, and one
for the provided message..  This prevents PRINTF_BUFR_SIZE from combining
the two messages into a single write.

The review https://reviews.freebsd.org/D16690 makes device_printf use
an sbuf w/ a printf drain to effectively buffer these messages together.

previously:
Trying to mount root from ufs:/dev/ufs/rootfs [rw]...
mmc0:  Instruction Set Attributes 0 = 
ACMD42 failed, RESULT: 4
 Instruction Set Attributes 1 = <>
mmc0:  Processor Features 0 = 
Card at relative address 43690 failed to set bus width

w/ patch:
mmc0: ACMD42 failed, RESULT: 4
Trying to mount root from ufs:/dev/ufs/rootfs [rw]...
 Instruction Set Attributes 0 = 
mmc0: Card at relative address 43690 failed to set bus width
 Instruction Set Attributes 1 = <>
 Processor Features 0 = 

Comments?

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: install: auto_master: No such file or directory

2018-08-07 Thread John-Mark Gurney
r/src
> > > *** Error code 1
> > >
> > > Stop.
> > > make[1]: stopped in /usr/src
> > > *** Error code 1
> > >
> > > Stop.
> > > make: stopped in /usr/src
> > > # less UPDATING
> > > # svn up
> > > Updating '.':
> > > At revision 337340.
> > >
> > >
> > > --
> > >  Lars Schotte
> > >  Mudro??ova 13
> > > 92101 Pieany
> > > ___
> > > freebsd-current@freebsd.org mailing list
> > > https://lists.freebsd.org/mailman/listinfo/freebsd-current
> > > To unsubscribe, send any mail to "freebsd-current-unsubscribe@
> > freebsd.org"
> >
> >
> > It's been on my wishlist that buildworld includes a 'fake installworld,
> > copying
> > /bin /usr/bin subtrees into a chroot or something... ' so if the
> > real one fails, where it installed to could be copied onto / ... with gcp
> > -R or some
> > such, and the procedure included in UPDATING.
> >
> >
> Alas this isn't possible to do in buildworld, because it would require root
> privileges.  buildworld needs to be done in environments where those
> privileges aren't available, like on universe12a.freebsd.org.

You can run installworld w/o root privs:
#   -DNO_ROOT install without using root privilege


-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: vlans + bridging is "interesting"

2017-11-24 Thread John-Mark Gurney
Rodney W. Grimes wrote this message on Fri, Nov 24, 2017 at 18:26 -0800:
> > I decided to try to run some bhyve VM's on my machine and bridge
> > them to a guest vlan on my main interface.  I also want to support
> > running bhyve VM's on the untagged part of the interface as well
> > (this is the key problem as I'll describe later).
> > 
> > I configure it as you'd expect.  Bridge the main interface em0, and
> > put the local IP's on the bridge0.  Then I added an interface em0.14
> > that untags packets from em0, and added it to bridge1 along w/ a tap0
> > for the VM.  This does not work.  Packet goes out and comes back and
> > is observed on em0, but never appears on either em0.14 or bridge1.
> > 
> > After seeing: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=139268
> > 
> > I decide to look on bridge0, and see the tagged vlan packet on that
> > interface.  I attempted to add bridge0 as the vlandev for em0.14, but
> > that doesn't work:
> > #ifconfig em0.14 vlan 14 vlandev bridge0
> > ifconfig: SIOCSETVLAN: Protocol not supported
> > 
> > So, I did finally get things working by using epair.  I added an epair
> > to the bridge, and that allows me to untag the packet, and pass on to
> > bridge1.
> > 
> > I have not attempted to use the patch in 139268, but if people think
> > it is an acceptable solution (with patch, if I set LINK0, it should work
> > w/ original configuration), I'll test and commit the patch.
> > 
> > Otherwise, please submit another fix.
> 
> I am also experiencing difficulties with vlan +briding +bhyve.  It
> seems the host that can talk just fine out a trunked em0 interface
> using vlan32 and vlan34 to all my other hardware can NOT talk to
> my bhyve guests.  Those bhyve guests can also talk out that
> same interface to other hardware, but they are being passed in
> the trunked interface, ie direct tap of bridge of em0 and the
> vlan tagging/untagging is being done inside the guest.
> 
> All the guests can talk to each other and they can all talk
> to real hardware that is via the em0 hardware, same for the
> host, but the host can not talk to the guests nor the guests
> to the host.

This is probably related.  I'm going to take a stab at your config,
so correct me if I'm wrong:
bridge0 w/ em0 & tapX
vlan32 on em0 vlan 32
vlan34 on em0 vlan 34

If this is the case, you're running into the same issue that I'm running
into...  The issue is that when a packet comes in on em0, it is
forwarded directly to bridge0, bypassing the vlan interfaces...

If you do what I did above, which is add an epair interface to the
bridge, and then add the vlan's off the epair interface, it will work...
As w/ the epair you are effectively simulating what your VM does, and
doing the encap/decap as the "VM" in the host...

The issue is that packets make it out em0 properly wrapped, but they'll
never make it back to the vlan32 interface, em0 forwards it directly to
bridge0, bypassing the vlan32 interface, and the bridge0 has no where to
deliver the packet, and it gets dropped...

The reason the vms work is that as you're doing decap in the VM, the
encapsulated packets are making it successfully to them, and their
encapsulated replies are making it safely back to the switch...

> My guess is that the arp's are not being seen by the bridge
> cause they are wrapping in vlan tags thus the bridge
> never learns all the mac addresses, but this is just a
> guess.

I finally figured this out w/ tcpdump, as tcpdump was showing the
packets going out em0.14 (in my case), but the reply was never making
it back to em0.14.  I was seeing it on em0 w/ "tcpdump -i em0 vlan 14",
and then, when I ran "tcpdump -i bridge0 vlan 14", I saw this missing
packet, which is how I decided to come up w/ the epair "solution"...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


vlans + bridging is "interesting"

2017-11-24 Thread John-Mark Gurney
Hello,

I decided to try to run some bhyve VM's on my machine and bridge
them to a guest vlan on my main interface.  I also want to support
running bhyve VM's on the untagged part of the interface as well
(this is the key problem as I'll describe later).

I configure it as you'd expect.  Bridge the main interface em0, and
put the local IP's on the bridge0.  Then I added an interface em0.14
that untags packets from em0, and added it to bridge1 along w/ a tap0
for the VM.  This does not work.  Packet goes out and comes back and
is observed on em0, but never appears on either em0.14 or bridge1.

After seeing: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=139268

I decide to look on bridge0, and see the tagged vlan packet on that
interface.  I attempted to add bridge0 as the vlandev for em0.14, but
that doesn't work:
#ifconfig em0.14 vlan 14 vlandev bridge0
ifconfig: SIOCSETVLAN: Protocol not supported

So, I did finally get things working by using epair.  I added an epair
to the bridge, and that allows me to untag the packet, and pass on to
bridge1.

I have not attempted to use the patch in 139268, but if people think
it is an acceptable solution (with patch, if I set LINK0, it should work
w/ original configuration), I'll test and commit the patch.

Otherwise, please submit another fix.

Thanks.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: OpenSSH HPN

2015-11-14 Thread John-Mark Gurney
   = 0x20:0x80963615
stack pointer   = 0x28:0xfe001de65580
frame pointer   = 0x28:0xfe001de655d0
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process = 0 (dummynet)
[ thread pid 0 tid 100084 ]
Stopped at  __mtx_lock_flags+0x55:  movq(%r13),%rax
db> bt
Tracing pid 0 tid 100084 td 0xf8000505c000
__mtx_lock_flags() at __mtx_lock_flags+0x55/frame 0xfe001de655d0
doselwakeup() at doselwakeup+0xb5/frame 0xfe001de65610
sowakeup() at sowakeup+0x3b/frame 0xfe001de65640
tcp_do_segment() at tcp_do_segment+0x26b9/frame 0xfe001de65730
tcp_input() at tcp_input+0xfb7/frame 0xfe001de65880
ip_input() at ip_input+0x175/frame 0xfe001de658e0
netisr_dispatch_src() at netisr_dispatch_src+0x86/frame 0xfe001de65950
dummynet_send() at dummynet_send+0x153/frame 0xfe001de65980
dummynet_task() at dummynet_task+0x2e3/frame 0xfe001de659e0
taskqueue_run_locked() at taskqueue_run_locked+0xf0/frame 0xfe001de65a40
taskqueue_thread_loop() at taskqueue_thread_loop+0x88/frame 0xfe001de65a70
fork_exit() at fork_exit+0x84/frame 0xfe001de65ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfe001de65ab0
--- trap 0, rip = 0, rsp = 0xfe001de65b70, rbp = 0 ---
db> 

I wasn't able to get a dump because my swap was misconfigured...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: OpenSSH HPN

2015-11-14 Thread John-Mark Gurney
Allan Jude wrote this message on Sat, Nov 14, 2015 at 11:53 -0500:
> On 2015-11-14 02:47, John-Mark Gurney wrote:
> > Allan Jude wrote this message on Thu, Nov 12, 2015 at 17:57 -0500:
> >> On 2015-11-12 12:56, John-Mark Gurney wrote:
> >>> Allan Jude wrote this message on Thu, Nov 12, 2015 at 12:15 -0500:
> >>>> On 2015-11-11 19:06, Slawa Olhovchenkov wrote:
> >>>>> On Wed, Nov 11, 2015 at 01:32:27PM -0800, Bryan Drewery wrote:
> >>>>>
> >>>>>> On 11/10/2015 1:42 AM, Dag-Erling Smørgrav wrote:
> >>>>>>>  I would also like to remove the NONE cipher
> >>>>>>> patch, which is also available in the port (off by default, just like 
> >>>>>>> in
> >>>>>>> base).
> >>>>>>
> >>>>>> Fun fact, it's been broken in the port for several months with no
> >>>>>> complaints. It was just reported and fixed upstream in the last day and
> >>>>>> I wrote in a similar fix in the port. That speaks a lot about its usage
> >>>>>> in the port currently.
> >>>>>
> >>>>> I am try using NPH/NONE with base ssh and confused: don't see
> >>>>> performance rise, too complex to enable and too complex for use.
> >>>>
> >>>> I did a few quick (and dirty) benchmarks and it shows that the NONE
> >>>> cipher definitely makes a difference. Version of OpenSSL also seems to
> >>>> make a difference, as one might expect.
> >>>>
> >>>> Note: openssh from ports seems to link against both base and ports
> >>>> libcrypto, I am still trying to make sure this isn't corrupting my
> >>>> benchmark results.
> >>>
> >>> You don't need the aesni.ko module loaded for OpenSSL (which is how
> >>> OpenSSH uses most crypto algos) to use AES-NI..
> >>>
> >>> Also, do you set any sysctl's to play w/ the buffer sizes or anything?
> >>>
> >>>> I am still debugging my dummynet setup to be able to prove that HPN
> >>>> makes a difference (but it does).
> >>>
> >>> Does my example on the page not work for you?
> >>>
> >>>> https://wiki.freebsd.org/SSHPerf
> >>>
> >>
> >> I found that when I set even 5ms of delay with dummynet, bandwidth over
> >> the LAN drops more than it should. Dummynet is limiting the rate rather
> >> than just adding the delay. I am investigating.
> >>
> >> I found this document:
> >> http://www.cs.unc.edu/~jeffay/dirt/FAQ/hstcp-howto.pdf
> >>
> >> Which is from the 6.x era, but suggests:
> >>
> >> "One subtle bug exists in the stock Dummynet implementation that should
> >> be corrected for experiments.  When a packet arrives in dummynet it is
> >> shoved into a queue which limits the bandwidth a TCP flow may use. Upon
> >> exit from the queue, the packet is transferred to a pipe where it sits
> >> for any configured amount of delay time and might possibly be dropped
> >> depending on the loss probability. Once the delay time has passed, the
> >> packet is released to ip output."
> >>
> >> May be the cause of my problem
> > 
> > Ahhh, probably need to adjust:
> > net.inet.ip.dummynet.pipe_byte_limit: 1048576
> > net.inet.ip.dummynet.pipe_slot_limit: 100
> > 
> > But even w/ the above limits and 5ms, you should still be able to push
> > 200MB/sec...
> 
> I worked with Hiren and some of his dtrace magic and figured out that
> dummynet was not my issue. I didn't end up needing to change the
> dummynet pipe slot/byte limit in order to get the full 10gig/sec even
> with 100ms delay from dummynet.
> 
> You just need to adjust:
> 
> net.inet.tcp.sendbuf_max=BDP
> net.inet.tcp.recvbuf_max=BDP
> 
> kern.ipc.maxsockbuf= ( BDP * (2048+256) ) / 2048
> 
> for a 50ms RTT:
> 
> BDP = 10gbps * .05 = ~60mb

I forgot to include _max adjustments on the page (the maxsockbuf was
there), but in all of my tests, I can't get close to that..  In my case,
I can demonstrate 20MB/sec+ over the link, and w/ a 100ms RTT, that'd
be a 2MB buffer size, and even when I increase these to 8MB, and
increase kern.ipc.maxsockbuf to 8MB too (otherwise _max is
meaningless), I still get 1.5MB/sec, not even close...

I do notice w/ nc that it'll slowly increase, and then suddenly back
off, just to closely increase again... I wonder if there is some issue
w/ TSO or tap that is causing issues...

> It can also greatly help to increase:
> net.inet.tcp.abc_l_var
> 
> Which is how many additional segments the CWNDs is incremented by each
> RTT during slow-start.
> 
> 
> I am still working on my set of benchmarks to show what different HPN
> makes with different RTT values, as well as what might be required to
> achieve max throughput for SSH over both LAN and the internet.
> 
> (For my company, we regularly transmit 500GB ZFS datasets over the
> public internet on 1gbps or 2x1gbps connections)

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: OpenSSH HPN

2015-11-13 Thread John-Mark Gurney
Allan Jude wrote this message on Thu, Nov 12, 2015 at 17:57 -0500:
> On 2015-11-12 12:56, John-Mark Gurney wrote:
> > Allan Jude wrote this message on Thu, Nov 12, 2015 at 12:15 -0500:
> >> On 2015-11-11 19:06, Slawa Olhovchenkov wrote:
> >>> On Wed, Nov 11, 2015 at 01:32:27PM -0800, Bryan Drewery wrote:
> >>>
> >>>> On 11/10/2015 1:42 AM, Dag-Erling Smørgrav wrote:
> >>>>>  I would also like to remove the NONE cipher
> >>>>> patch, which is also available in the port (off by default, just like in
> >>>>> base).
> >>>>
> >>>> Fun fact, it's been broken in the port for several months with no
> >>>> complaints. It was just reported and fixed upstream in the last day and
> >>>> I wrote in a similar fix in the port. That speaks a lot about its usage
> >>>> in the port currently.
> >>>
> >>> I am try using NPH/NONE with base ssh and confused: don't see
> >>> performance rise, too complex to enable and too complex for use.
> >>
> >> I did a few quick (and dirty) benchmarks and it shows that the NONE
> >> cipher definitely makes a difference. Version of OpenSSL also seems to
> >> make a difference, as one might expect.
> >>
> >> Note: openssh from ports seems to link against both base and ports
> >> libcrypto, I am still trying to make sure this isn't corrupting my
> >> benchmark results.
> > 
> > You don't need the aesni.ko module loaded for OpenSSL (which is how
> > OpenSSH uses most crypto algos) to use AES-NI..
> > 
> > Also, do you set any sysctl's to play w/ the buffer sizes or anything?
> > 
> >> I am still debugging my dummynet setup to be able to prove that HPN
> >> makes a difference (but it does).
> > 
> > Does my example on the page not work for you?
> > 
> >> https://wiki.freebsd.org/SSHPerf
> > 
> 
> I found that when I set even 5ms of delay with dummynet, bandwidth over
> the LAN drops more than it should. Dummynet is limiting the rate rather
> than just adding the delay. I am investigating.
> 
> I found this document:
> http://www.cs.unc.edu/~jeffay/dirt/FAQ/hstcp-howto.pdf
> 
> Which is from the 6.x era, but suggests:
> 
> "One subtle bug exists in the stock Dummynet implementation that should
> be corrected for experiments.  When a packet arrives in dummynet it is
> shoved into a queue which limits the bandwidth a TCP flow may use. Upon
> exit from the queue, the packet is transferred to a pipe where it sits
> for any configured amount of delay time and might possibly be dropped
> depending on the loss probability. Once the delay time has passed, the
> packet is released to ip output."
> 
> May be the cause of my problem

Ahhh, probably need to adjust:
net.inet.ip.dummynet.pipe_byte_limit: 1048576
net.inet.ip.dummynet.pipe_slot_limit: 100

But even w/ the above limits and 5ms, you should still be able to push
200MB/sec...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: OpenSSH HPN

2015-11-12 Thread John-Mark Gurney
Allan Jude wrote this message on Thu, Nov 12, 2015 at 12:15 -0500:
> On 2015-11-11 19:06, Slawa Olhovchenkov wrote:
> > On Wed, Nov 11, 2015 at 01:32:27PM -0800, Bryan Drewery wrote:
> > 
> >> On 11/10/2015 1:42 AM, Dag-Erling Smørgrav wrote:
> >>>  I would also like to remove the NONE cipher
> >>> patch, which is also available in the port (off by default, just like in
> >>> base).
> >>
> >> Fun fact, it's been broken in the port for several months with no
> >> complaints. It was just reported and fixed upstream in the last day and
> >> I wrote in a similar fix in the port. That speaks a lot about its usage
> >> in the port currently.
> > 
> > I am try using NPH/NONE with base ssh and confused: don't see
> > performance rise, too complex to enable and too complex for use.
> 
> I did a few quick (and dirty) benchmarks and it shows that the NONE
> cipher definitely makes a difference. Version of OpenSSL also seems to
> make a difference, as one might expect.
> 
> Note: openssh from ports seems to link against both base and ports
> libcrypto, I am still trying to make sure this isn't corrupting my
> benchmark results.

You don't need the aesni.ko module loaded for OpenSSL (which is how
OpenSSH uses most crypto algos) to use AES-NI..

Also, do you set any sysctl's to play w/ the buffer sizes or anything?

> I am still debugging my dummynet setup to be able to prove that HPN
> makes a difference (but it does).

Does my example on the page not work for you?

> https://wiki.freebsd.org/SSHPerf

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: OpenSSH HPN

2015-11-11 Thread John-Mark Gurney
Ben Woods wrote this message on Wed, Nov 11, 2015 at 16:27 +0800:
> On Wednesday, 11 November 2015, John-Mark Gurney <j...@funkthat.com> wrote:
> 
> > Ben Woods wrote this message on Wed, Nov 11, 2015 at 15:40 +0800:
> > > I have to agree that there are cases when the NONE cipher makes sense,
> > and
> > > it is up to the end user to make sure they know what they are doing.
> > >
> > > Personally I have used it at home to backup my old FreeBSD server (which
> > > does not have AESNI) over a dedicated network connection to a backup
> > server
> > > using rsync/ssh. Since it was not possible for anyone else to be on that
> > > local network, and the server was so old it didn't have AESNI and would
> > > soon be retired, using the NONE cipher sped up the transfer
> > significantly.
> >
> > If you have a trusted network, why not just use nc?
> 
> Honest answer: ignorance of how I can use netcat together with rsync.

A quick google of rsync nc, turned up method 2 & 4 from:
https://rsync.samba.org/firewall.html

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: OpenSSH HPN

2015-11-11 Thread John-Mark Gurney
Daniel Kalchev wrote this message on Wed, Nov 11, 2015 at 17:49 +0200:
> It is my understanding, that using the NONE cypher is not identical to using 
> ???the old tools??? (rsh/rlogin/rcp).
> 
> When ssh uses the NONE cypher, credentials and authorization are still 
> encrypted and verified. Only the actual data payload is not encrypted.

Except the point is that you ALREADY trust your network, so you don't
need to encrypt the credentials and authorizations, otherwise, why are
you running unencrypted payloads?

In fact, if you aren't running at least a MAC, or a final verify, and
you're transfering large amounts of data (multiple gigabytes), the data
can and will likely be corrupted...

See:
http://noahdavids.org/self_published/CRC_and_checksum.html

Having not used the NONE cipher, I don't know if the MAC is also
removed or not...  Either way, the MAC is still the long poll when
it comes to encryption w/ AES-NI...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: OpenSSH HPN

2015-11-11 Thread John-Mark Gurney
Ben Woods wrote this message on Wed, Nov 11, 2015 at 15:40 +0800:
> On Wednesday, 11 November 2015, Bryan Drewery <bdrew...@freebsd.org> wrote:
> 
> > On 11/10/15 9:52 AM, John-Mark Gurney wrote:
> > > My vote is to remove the HPN patches.  First, the NONE cipher made more
> > > sense back when we didn't have AES-NI widely available, and you were
> > > seriously limited by it's performance.  Now we have both aes-gcm and
> > > chacha-poly which it's performance should be more than acceptable for
> > > today's uses (i.e. cipher performance is 2GB/sec+).
> >
> > AES-NI doesn't help the absurdity of double-encrypting when using scp or
> > rsync/ssh over an encrypted VPN, which is where NONE makes sense to use
> > for me.
> 
> I have to agree that there are cases when the NONE cipher makes sense, and
> it is up to the end user to make sure they know what they are doing.
> 
> Personally I have used it at home to backup my old FreeBSD server (which
> does not have AESNI) over a dedicated network connection to a backup server
> using rsync/ssh. Since it was not possible for anyone else to be on that
> local network, and the server was so old it didn't have AESNI and would
> soon be retired, using the NONE cipher sped up the transfer significantly.

If you have a trusted network, why not just use nc?

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: OpenSSH HPN

2015-11-10 Thread John-Mark Gurney
Dag-Erling Smrgrav wrote this message on Tue, Nov 10, 2015 at 10:42 +0100:
> Therefore, I would like to remove the HPN patches from base and refer
> anyone who really needs them to the openssh-portable port, which has
> them as a default option.  I would also like to remove the NONE cipher
> patch, which is also available in the port (off by default, just like in
> base).

My vote is to remove the HPN patches.  First, the NONE cipher made more
sense back when we didn't have AES-NI widely available, and you were
seriously limited by it's performance.  Now we have both aes-gcm and
chacha-poly which it's performance should be more than acceptable for
today's uses (i.e. cipher performance is 2GB/sec+).

Second, I did some testing recently due to a thread on -net, and I
found no significant (not run statistically though) difference in
performance between in HEAD ssh and OpenSSH 7.1p1.  I started a wiki
page to talk about this:
https://wiki.freebsd.org/SSHPerf

Feel free to add to the page any more info.

There are other apparent issues w/ ssh that keeps it's performance low
on high latency links, but I haven't spend the time to figure out what
they are, but in my testing HPN did not increase performance to make
use of the fat but high latency link.

So, if it's not increasing performance and making us fall behind, why
bother with the trouble of keeping the patch?

If someone is willing to spend the time doing benchmarks, and prove
that the HPN patches do make a difference, I'm willing to work with
them to figure out why my tests didn't work and change my vote.  I
also believe that the defaults should be enough, if you have to tune
or enable features, then you can install from ports or compile yourself.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: OpenSSH HPN

2015-11-10 Thread John-Mark Gurney
Bryan Drewery wrote this message on Tue, Nov 10, 2015 at 16:32 -0800:
> On 11/10/15 9:52 AM, John-Mark Gurney wrote:
> > My vote is to remove the HPN patches.  First, the NONE cipher made more
> > sense back when we didn't have AES-NI widely available, and you were
> > seriously limited by it's performance.  Now we have both aes-gcm and
> > chacha-poly which it's performance should be more than acceptable for
> > today's uses (i.e. cipher performance is 2GB/sec+).
> 
> AES-NI doesn't help the absurdity of double-encrypting when using scp or
> rsync/ssh over an encrypted VPN, which is where NONE makes sense to use
> for me.

Different layers of protection...

Do you disable all encryption when you're transiting trusted networks
like your VPN?  If you don't, why is that ssh session so special?

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Timing issue with Dummynet on high kernel timer interrupt

2015-11-06 Thread John-Mark Gurney
Adrian Chadd wrote this message on Fri, Nov 06, 2015 at 11:15 -0800:
> Ideally there'd be both behaviours:
> 
> * You'd specify whether a timer/sleep needs to be exact or can
> withstand some jitter (which is what linux provides); and

Isn't that what the precision argument in callout is for?

See callout_reset_sbt(9):
 The sbt, pr, and flags arguments provide more control over the scheduled
 time including support for higher resolution times, specifying the
 precision of the scheduled time, and setting an absolute deadline instead
 of a relative timeout.  The callout is scheduled to execute in a time
 window which begins at the time specified in sbt and extends for the
 amount of time specified in pr.  If sbt specifies a time in the past, the
 window is adjusted to start at the current time.  A non-zero value for pr
 allows the callout subsystem to coalesce callouts scheduled close to each
 other into fewer timer interrupts, reducing processing overhead and power
 consumption.  These flags may be specified to adjust the interpretation
 of sbt and pr:


-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Depreciate and remove gbde

2015-10-29 Thread John-Mark Gurney
Lyndon Nerenberg wrote this message on Mon, Oct 26, 2015 at 19:06 -0700:
> On Oct 24, 2015, at 12:06 PM, John-Mark Gurney <j...@funkthat.com> wrote:
> 
> > The thing I like most about encryption is that when I RMA a bad
> > drive, I don't have to worry about my data leaking if I am unable
> > to overwrite all the data...
> 
> You are optimistic if you believe that.  We ($WORK) factor the cost of 
> DOA/warranty drives into our operational budget.  They never get RMAed.  We 
> drill them when they die.

Being a personal user, and having close to a 10% RMA rate on recent
hard drives, that would be a bit costly...

I consider a HD defective if it's under waranty and it's performance
drops below 80% of new, i.e. 130MB/sec normal sequential write drops
below 100MB/sec..

The weekest point is the passphrase/passfile protecting the master
key... In my case, I use a random passfile for these drives...  If
someone is able to break the passfile, or the AES-256 encryption, then
they must really want my data...  It'd be easier, even for governments,
to do a black bag job than recover partial data (it's one drive of a
RAIDZ array)...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Depreciate and remove gbde

2015-10-24 Thread John-Mark Gurney
Julian H. Stacey wrote this message on Sat, Oct 24, 2015 at 17:58 +0200:
> > >If you want a secure filesystem I think that at this particular time
> > >it would be entirely reasonable to use both gbde and geli stacked on
> > >top of each other[...]
> 
> I've often wondered if multiple encryption (CPU permitting) is sensible in 
> case one day some method is cracked but another stays secure.

Depends if you care about performance or not.  gbde is very slow, maybe
150MB/sec/core on a decently fast processor...  Where as geli is
~1GB/sec/core (AES-XTS)...

> There's been recent discussions on cracking algorithms at
>  http://lists.gnupg.org/pipermail/gnupg-users/2015-October/054586.html
> 
> I see man geli has:
>   Supports many cryptographic algorithms (currently AES-XTS,
>   AES-CBC, Blowfish-CBC, Camellia-CBC and 3DES-CBC).
> NAME section of man 1 gbde & geli both ref. GEOM.
> Skimming man 1 4 8 gbde geom I'm not sure how gbde compares.

gbde uses AES128-CBC, which is bad for modern processors that have AES-NI
instructions, as AES-CBC cannot be pipelined.

> > Nobody is going to break through the GELI or GBDE crypto, they'll
> > find their way to the keys instead, or more likely, jail you until
> > you sing.
> 
> Yes, if 'they' are physicaly present government, criminals etc.
> 
> Encryption (& perhaps multiple encryption) is nice against eg

The thing I like most about encryption is that when I RMA a bad
drive, I don't have to worry about my data leaking if I am unable
to overwrite all the data...  Also, for SSD's, where a complete
overwrite will not overwrite all the data, this helps that..

Note that even w/ drives purporting to provide hardware encryption
they don't do it very well:
http://arstechnica.com/security/2015/10/western-digital-self-encrypting-hard-drives-riddled-with-security-flaws/

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: is building kernel in /sys/amd64/conf depreciated in 11 ?

2015-10-21 Thread John-Mark Gurney
John wrote this message on Tue, Oct 20, 2015 at 17:40 +0100:
> as subject - is building kernel in /sys/amd64/conf depreciated?
> 
> I can get a modified kernel to build and install in /usr/src but not
> in /sys/amd64/conf. I always used to be able to do this in there, then
> again I either used -RELEASE or -STABLE. I used to do it like this:
> 
> 1. cd /sys/amd64/conf
> 2. cp GENERIC MYKERNEL
> 3. [make changes to MYKERNEL and save]
> 4. config MYKERNEL
> 5. cd ../compile/MYKERNEL
> 6. make cleandepend && make depend && make
> 
> This fails *every time* during make. It fails at this point:
> 
> Make[1]: "/storage/usr/ports/Mk/bsd.port.mk" line 1204: UNAME_r
> (11.0-CURRENT) and OSVERSION () do not agree on major version number.
  

You're trying to build a 11-CURRENT kernel on a 10-something userland
from the looks of it, and that has never been supported, it may work,
but when it doesn't, FreeBSD won't fix it...

If you do like building kernels the above way, you can do:
cd /usr/src
make kernel-toolchain
make buildenv   # which launches a shell
cd /amd64/conf
... traditional build method ...

The kernel-toolchain/buildenv builds the tools and sets up the
environment just like buildkernel does for the kernel compiles...

Or you need to update your compile box's userland to match the kernel
version that you're building...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Depreciate and remove gbde

2015-10-19 Thread John-Mark Gurney
O. Hartmann wrote this message on Mon, Oct 19, 2015 at 06:19 +0200:
> For me, I'd like to know what is the benefit/performance of each technique and
> a clear preparation of each ones advantages over the other. That would make 
> the
> decission process much easier and hopefully would not scare people away and
> announce "FreeBSD does not have a, b, c, ..." ...

So, one thing that the docs talk about is that geli uses the crypto(9)
framework.  This doesn't mean much on it's own, but if you have a machine
with AES-NI instructions or an accelerator card that supports the cipher
mode used, then you can get faster performance of hardware off load,
while gbde uses the software only routines which are slow..

I have put work into making AES-XTS very fast on AES-NI capable
machines...  On my test machine, I get about 1GB/sec on gzero... This
is close to real world (assuming infitely fast disc) vs. just running
the algorithm and posting those results (which result in 2GB/sec+ on
the same machine)...  You will not be able to achive that level of
performance w/ gbde.

Also, gbde uses CBC, while having some better crypto properties than
XTS, would require significant rewrite of gbde to make it perform...

I just noticed that the handbook also fails to mention that geli has
a mode that will verify the integrity of data which gbde does not have..

As we have discovered, if you can't authenticate your data, you really
can't trust it...  I personally have decided that I will use ZFS's sha256
checksums of the data as my integrity protection mechanism..  It is
highly unlikely that an attacker would be able to corrupt two AES-XTS
blocks to cause the sha256 checksum to match what they corrupted other
blocks to become...

So, in this reguard, if you run gbde w/ ZFS w/ sha256 checksums, then
are equivalent (besides the performance difference)...

I personally run geli encryption on my 8 drive ZFS array at home.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Depreciate and remove gbde

2015-10-19 Thread John-Mark Gurney
Ed Maste wrote this message on Mon, Oct 19, 2015 at 17:13 -0400:
> On 19 October 2015 at 16:50, John-Mark Gurney <j...@funkthat.com> wrote:
> > O. Hartmann wrote this message on Mon, Oct 19, 2015 at 06:19 +0200:
> >> For me, I'd like to know what is the benefit/performance of each technique 
> >> and
> >> a clear preparation of each ones advantages over the other. That would 
> >> make the
> >> decission process much easier and hopefully would not scare people away and
> >> announce "FreeBSD does not have a, b, c, ..." ...
> >
> > So, one thing that the docs talk about is that geli uses the crypto(9)
> > framework.  This doesn't mean much on it's own, but if you have a machine
> > with AES-NI instructions or an accelerator card that supports the cipher
> > mode used, then you can get faster performance of hardware off load,
> > while gbde uses the software only routines which are slow..
> 
> John-Mark, thanks for listing these differences. This is the sort of
> information we should have available for end users to help choose one
> or the other -- this info ought to make it into the handbook.

I'm working on updating the section now...

Also realized we should include verbage to say that it's best to use
page size sectors when possible to reduce overhead of the crypto...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Instant panic while trying run ports-mgmt/poudriere

2015-09-01 Thread John-Mark Gurney
Konstantin Belousov wrote this message on Tue, Sep 01, 2015 at 21:44 +0300:
> On Tue, Sep 01, 2015 at 11:24:06AM -0700, John-Mark Gurney wrote:
> > Andriy Gapon wrote this message on Thu, Aug 27, 2015 at 23:21 +0300:
> > > On 27/08/2015 21:09, John-Mark Gurney wrote:
> > > > Andriy Gapon wrote this message on Thu, Aug 27, 2015 at 10:21 +0300:
> > > >> On 27/08/2015 02:36, John-Mark Gurney wrote:
> > > >>> We should/cannot get here w/ an empty list.  If we do, then there is
> > > >>> something seriously wrong...  The current kn (which we must have as we
> > > >>> are here) MUST be on the list, but as you just showed, there are no
> > > >>> knotes on the list.
> > > >>>
> > > >>> Can you get me a print of the knote?  That way I can see what flags
> > > >>> are on it?
> > > >>
> > > >> Apologies if the following might sound a little bit patronizing, but it
> > > >> seems that you have got all the facts correctly, but somehow the
> > > >> connection between them did not become clear.
> > > >>
> > > >> So:
> > > >> 1. The list originally is NOT empty.  I guess that it has one entry, 
> > > >> but
> > > >> that's an unimportant detail.
> > > >> 2. This is why the loop is entered. It's a fact that it is entered.
> > > >> 3. The list becomes empty precisely because the entry is removed during
> > > >> the iteration in the loop (as kib has explained).  It's a fact that the
> > > >> list became empty at least in the panic that I reported.
> > > > 
> > > > On you're latest dump, you said:
> > > > Here is another +1 with r286922.
> > > > 
> > > > I can add a couple of bits of debugging data:   
> > > > 
> > > > 
> > > > 
> > > > (kgdb) fr 8 
> > > > 
> > > > #8  0x80639d60 in knote (list=0xf8019a733ea0,   
> > > > 
> > > > hint=2147483648, lockflags=) at
> > > > 
> > > > /usr/src/sys/kern/kern_event.c:1964 
> > > > 
> > > > 1964} else if ((lockflags & KNF_NOKQLOCK) != 0) {   
> > > > 
> > > > 
> > > > First off, that can't be r286922, per:
> > > > https://svnweb.freebsd.org/base/stable/10/sys/kern/kern_event.c?annotate=286922
> > > > 
> > > > line 1964 is blank...  The line of code above should be at line 1884,
> > > > so not sure what is wrong here...
> > > 
> > > No, it can not be indeed, because I am running head.
> > > r286922 was the latest version of the repository, not the head branch,
> > > at the moment when I pulled the repository via git.
> > > 
> > > > Assuming that the pc really is at the line, f_event has not yet been
> > > > called,
> > > 
> > > Even on the second loop iteration?
> > > 
> > > >which is why I said that the list cannot be empty yet, as
> > > > f_event hasn't been called yet to remove the knote...  It could be that
> > > > optimization moved stuff around, but if that is the case, then the
> > > > above wasn't useful..
> > > 
> > > I provided the disassembly of the code as well, it's very obvious how
> > > the code was translated.
> > > 
> > > >> 4. The element is not only unlinked from the list, but its memory is
> > > >> also freed.
> > > > 
> > > > Where is the memory freed?  A knote MUST NOT be freed in an f_event
> > > > handler.  The only location that a list element is allowed to be
> > > > freed is in knote_drop, which must happen after f_detach is called,
> > > > but that can't/won't happen from knote (I believe the timer handles
> > > > this specially, but we are talking about normal knlist type filters)..
> > > 
> > > Well, right.  knote()->filt_proc()->knlist_remove_inevent() just removes
> > > the knote from the list.  But then there is KNOTE_ACTIVATE() that passes
> > > the knote to a different owner (so to say). And given that the knote has
> > > EV

Re: Instant panic while trying run ports-mgmt/poudriere

2015-09-01 Thread John-Mark Gurney
Andriy Gapon wrote this message on Thu, Aug 27, 2015 at 23:21 +0300:
> On 27/08/2015 21:09, John-Mark Gurney wrote:
> > Andriy Gapon wrote this message on Thu, Aug 27, 2015 at 10:21 +0300:
> >> On 27/08/2015 02:36, John-Mark Gurney wrote:
> >>> We should/cannot get here w/ an empty list.  If we do, then there is
> >>> something seriously wrong...  The current kn (which we must have as we
> >>> are here) MUST be on the list, but as you just showed, there are no
> >>> knotes on the list.
> >>>
> >>> Can you get me a print of the knote?  That way I can see what flags
> >>> are on it?
> >>
> >> Apologies if the following might sound a little bit patronizing, but it
> >> seems that you have got all the facts correctly, but somehow the
> >> connection between them did not become clear.
> >>
> >> So:
> >> 1. The list originally is NOT empty.  I guess that it has one entry, but
> >> that's an unimportant detail.
> >> 2. This is why the loop is entered. It's a fact that it is entered.
> >> 3. The list becomes empty precisely because the entry is removed during
> >> the iteration in the loop (as kib has explained).  It's a fact that the
> >> list became empty at least in the panic that I reported.
> > 
> > On you're latest dump, you said:
> > Here is another +1 with r286922.
> > 
> > I can add a couple of bits of debugging data:   
> > 
> > 
> > 
> > (kgdb) fr 8 
> > 
> > #8  0x80639d60 in knote (list=0xf8019a733ea0,   
> > 
> > hint=2147483648, lockflags=) at
> > 
> > /usr/src/sys/kern/kern_event.c:1964 
> > 
> > 1964} else if ((lockflags & KNF_NOKQLOCK) != 0) {   
> > 
> > 
> > First off, that can't be r286922, per:
> > https://svnweb.freebsd.org/base/stable/10/sys/kern/kern_event.c?annotate=286922
> > 
> > line 1964 is blank...  The line of code above should be at line 1884,
> > so not sure what is wrong here...
> 
> No, it can not be indeed, because I am running head.
> r286922 was the latest version of the repository, not the head branch,
> at the moment when I pulled the repository via git.
> 
> > Assuming that the pc really is at the line, f_event has not yet been
> > called,
> 
> Even on the second loop iteration?
> 
> >which is why I said that the list cannot be empty yet, as
> > f_event hasn't been called yet to remove the knote...  It could be that
> > optimization moved stuff around, but if that is the case, then the
> > above wasn't useful..
> 
> I provided the disassembly of the code as well, it's very obvious how
> the code was translated.
> 
> >> 4. The element is not only unlinked from the list, but its memory is
> >> also freed.
> > 
> > Where is the memory freed?  A knote MUST NOT be freed in an f_event
> > handler.  The only location that a list element is allowed to be
> > freed is in knote_drop, which must happen after f_detach is called,
> > but that can't/won't happen from knote (I believe the timer handles
> > this specially, but we are talking about normal knlist type filters)..
> 
> Well, right.  knote()->filt_proc()->knlist_remove_inevent() just removes
> the knote from the list.  But then there is KNOTE_ACTIVATE() that passes
> the knote to a different owner (so to say). And given that the knote has
> EV_ONESHOT set on it (in filt_proc) and that poudriere can put quite a
> stress load on a system, I am not surprised that another thread gets a
> chance to call knote_drop() on the knote before the original thread
> proceeds to the next iteration.

Ok, I think I have identified the race that you guys were trying to
tell me about, and though the _SAFE macro would be a similar fix, I'm
going to rewrite the loop so that this is more explicit on what
is happening here...

So, the race is this...  In knote, when the note is removed by
f_event, things are find until the KQ lock is dropped...  Once this
lock is dropped, effective ownership of the knote is transfered
from the knlist to the kq lock as the _DETACHED flag is now set,
which means that reading any fields from that note is undefined..

Once the kq lock is released in knote, then it is possible for a
functional like kqueue_scan to endup knote_drop'ing the note...

Upon further e

Re: Instant panic while trying run ports-mgmt/poudriere

2015-08-27 Thread John-Mark Gurney
Andriy Gapon wrote this message on Thu, Aug 27, 2015 at 10:21 +0300:
 On 27/08/2015 02:36, John-Mark Gurney wrote:
  We should/cannot get here w/ an empty list.  If we do, then there is
  something seriously wrong...  The current kn (which we must have as we
  are here) MUST be on the list, but as you just showed, there are no
  knotes on the list.
  
  Can you get me a print of the knote?  That way I can see what flags
  are on it?
 
 Apologies if the following might sound a little bit patronizing, but it
 seems that you have got all the facts correctly, but somehow the
 connection between them did not become clear.
 
 So:
 1. The list originally is NOT empty.  I guess that it has one entry, but
 that's an unimportant detail.
 2. This is why the loop is entered. It's a fact that it is entered.
 3. The list becomes empty precisely because the entry is removed during
 the iteration in the loop (as kib has explained).  It's a fact that the
 list became empty at least in the panic that I reported.

On you're latest dump, you said:
Here is another +1 with r286922.
I can add a couple of bits of debugging data:   

(kgdb) fr 8 
#8  0x80639d60 in knote (list=0xf8019a733ea0,   
hint=2147483648, lockflags=value optimized out) at
/usr/src/sys/kern/kern_event.c:1964 
1964} else if ((lockflags  KNF_NOKQLOCK) != 0) {   

First off, that can't be r286922, per:
https://svnweb.freebsd.org/base/stable/10/sys/kern/kern_event.c?annotate=286922

line 1964 is blank...  The line of code above should be at line 1884,
so not sure what is wrong here...

Assuming that the pc really is at the line, f_event has not yet been
called, which is why I said that the list cannot be empty yet, as
f_event hasn't been called yet to remove the knote...  It could be that
optimization moved stuff around, but if that is the case, then the
above wasn't useful..

 4. The element is not only unlinked from the list, but its memory is
 also freed.

Where is the memory freed?  A knote MUST NOT be freed in an f_event
handler.  The only location that a list element is allowed to be
freed is in knote_drop, which must happen after f_detach is called,
but that can't/won't happen from knote (I believe the timer handles
this specially, but we are talking about normal knlist type filters)..

The rest of your explination is invalid due to the invalid assumption
of this point...

If you can provide to me where the knote is free'd in knote, w/
function/line number stack trace (does not have to be dump, but a
sample call path), then I'll reconsider, and fix that bug...
 5. That's why we have the use after free: SLIST_FOREACH is trying to get
 a pointer to a next element from the freed memory.
 6. This is why the commit for trashing the freed memory made all the
 difference: previously the freed memory was unlikely to be re-used /
 modified, so the use-after-free had a high chance of succeeding.  It's a
 fact that in my panic there was an attempt to dereference a trashed pointer.
 7. Finally, this is why SLIST_FOREACH_SAFE helps here: we stash the
 pointer to the next element beforehand and, thus, we do not access the
 freed memory.
 
 Please let me know if you see any fault in above reasoning or if
 something is still no clear.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Instant panic while trying run ports-mgmt/poudriere

2015-08-26 Thread John-Mark Gurney
Andriy Gapon wrote this message on Sun, Aug 23, 2015 at 09:54 +0300:
 On 12/08/2015 17:11, Lawrence Stewart wrote:
  On 08/07/15 07:33, Pawel Pekala wrote:
  Hi K.,
 
  On 2015-08-06 12:33 -0700, K. Macy km...@freebsd.org wrote:
  Is this still happening?
 
  Still crashes:
  
  +1 for me running r286617
 
 Here is another +1 with r286922.
 I can add a couple of bits of debugging data:
 
 (kgdb) fr 8
 #8  0x80639d60 in knote (list=0xf8019a733ea0,
 hint=2147483648, lockflags=value optimized out) at
 /usr/src/sys/kern/kern_event.c:1964
 1964} else if ((lockflags  KNF_NOKQLOCK) != 0) {
 (kgdb) p *list
 $2 = {kl_list = {slh_first = 0x0}, kl_lock = 0x8063a1e0

We should/cannot get here w/ an empty list.  If we do, then there is
something seriously wrong...  The current kn (which we must have as we
are here) MUST be on the list, but as you just showed, there are no
knotes on the list.

Can you get me a print of the knote?  That way I can see what flags
are on it?

 knlist_mtx_lock, kl_unlock = 0x8063a200 knlist_mtx_unlock,
   kl_assert_locked = 0x8063a220 knlist_mtx_assert_locked,
 kl_assert_unlocked = 0x8063a240 knlist_mtx_assert_unlocked,
   kl_lockarg = 0xf8019a733bb0}
 (kgdb) disassemble
 Dump of assembler code for function knote:
 0x80639d00 knote+0:   push   %rbp
 0x80639d01 knote+1:   mov%rsp,%rbp
 0x80639d04 knote+4:   push   %r15
 0x80639d06 knote+6:   push   %r14
 0x80639d08 knote+8:   push   %r13
 0x80639d0a knote+10:  push   %r12
 0x80639d0c knote+12:  push   %rbx
 0x80639d0d knote+13:  sub$0x18,%rsp
 0x80639d11 knote+17:  mov%edx,%r12d
 0x80639d14 knote+20:  mov%rsi,-0x30(%rbp)
 0x80639d18 knote+24:  mov%rdi,%rbx
 0x80639d1b knote+27:  test   %rbx,%rbx
 0x80639d1e knote+30:  je 0x80639ef6 knote+502
 0x80639d24 knote+36:  mov%r12d,%eax
 0x80639d27 knote+39:  and$0x1,%eax
 0x80639d2a knote+42:  mov%eax,-0x3c(%rbp)
 0x80639d2d knote+45:  mov0x28(%rbx),%rdi
 0x80639d31 knote+49:  je 0x80639d38 knote+56
 0x80639d33 knote+51:  callq  *0x18(%rbx)
 0x80639d36 knote+54:  jmp0x80639d42 knote+66
 0x80639d38 knote+56:  callq  *0x20(%rbx)
 0x80639d3b knote+59:  mov0x28(%rbx),%rdi
 0x80639d3f knote+63:  callq  *0x8(%rbx)
 0x80639d42 knote+66:  mov%rbx,-0x38(%rbp)
 0x80639d46 knote+70:  mov(%rbx),%rbx
 0x80639d49 knote+73:  test   %rbx,%rbx
 0x80639d4c knote+76:  je 0x80639ee5 knote+485
 0x80639d52 knote+82:  and$0x2,%r12d
 0x80639d56 knote+86:  nopw   %cs:0x0(%rax,%rax,1)
 0x80639d60 knote+96:  mov0x28(%rbx),%r14
 
 Panic is in the last quoted instruction.
 And:
 (kgdb) i reg
 rax0x246582
 rbx0xdeadc0dedeadc0de   -2401050962867404578
 rcx0x0  0
 rdx0x12e302
 rsi0x80a26a5a   -2136839590
 rdi0x80e81b80   -2132272256
 rbp0xfe02b7efea20   0xfe02b7efea20
 rsp0xfe02b7efe9e0   0xfe02b7efe9e0
 r8 0x80a269ce   -2136839730
 r9 0x80e82838   -2132269000
 r100x1  65536
 r110x80fabd10   -2131051248
 r120x0  0
 r130xf801ff84a818   -8787511171048
 r140xf801ff84a800   -8787511171072
 r150xf8019a6974f0   -8789207452432
 rip0x80639d60   0x80639d60 knote+96
 eflags 0x10286  66182
 
 I think that $rbx stands out here (this is a kernel with INVARIANTS).

Yeh, it was probably r284861 that I added to catch use after free bugs
like this...  You could try reverting r284861 to see if the bug goes
away... If it does, then this is most likely a use after free bug...

 Looking at the code, is it possible that one of the calls from within
 the loop's body modifies the list?  If that is so and provided that is a
 valid behavior, then maybe using SLIST_FOREACH_SAFE would help.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Instant panic while trying run ports-mgmt/poudriere

2015-08-26 Thread John-Mark Gurney
Konstantin Belousov wrote this message on Mon, Aug 24, 2015 at 11:10 +0300:
 On Sun, Aug 23, 2015 at 10:35:44PM -0700, John-Mark Gurney wrote:
  Konstantin Belousov wrote this message on Sun, Aug 23, 2015 at 15:54 +0300:
 if (kev-flags  EV_ADD)
   - tkn = knote_alloc(waitok);  /* prevent waiting with locks */
   + /*
   +  * Prevent waiting with locks.  Non-sleepable
   +  * allocation failures are handled in the loop, only
   +  * if the spare knote appears to be actually required.
   +  */
   + tkn = knote_alloc(waitok);
  
  if you add this comment, please add curly braces around the block...
 Ok.
 
  
 else
 tkn = NULL;

   @@ -1310,8 +1315,7 @@ done:
 FILEDESC_XUNLOCK(td-td_proc-p_fd);
 if (fp != NULL)
 fdrop(fp, td);
   - if (tkn != NULL)
   - knote_free(tkn);
   + knote_free(tkn);
  
  Probably should just change knote_free to a static inline that does
  a uma_zfree as uma_zfree also does nothing is the input is NULL...
 This was already done in the patch (the removal of the NULL check in
 knote_free()). I usually do not add excessive inline keywords. Compilers
 are good, sometimes even too good, at figuring out the possibilities for
 inlining. knote_free() is inlined automatically.

Though it is, if we really change knote_free to a bare uma_free, then
either mark it inline (to be explicit about it's behavior), or make a
macro out of it... I don't particularly like functions that contain one
line of simple code...

   @@ -1948,7 +1948,7 @@ knote(struct knlist *list, long hint, int lockflags)
  * only safe if you want to remove the current item, which we are
  * not doing.
  */
   - SLIST_FOREACH(kn, list-kl_list, kn_selnext) {
   + SLIST_FOREACH_SAFE(kn, list-kl_list, kn_selnext, tkn) {
  
  Clearly you didn't read the comment that preceeds this line, or at
  least didn't update it:
   * SLIST_FOREACH, SLIST_FOREACH_SAFE is not safe in our case, it is
   * only safe if you want to remove the current item, which we are
   * not doing.
  
  So, you'll need to be more specific in why this needs to change...
  When I wrote this code, I spent a lot of time looking at this, and
  reasoned as to why SLIST_FOREACH_SAFE was NOT correct usage here...
 I explained what happens in the message.  The knote list is modified
 by the filter, see knlist_remove_inevent() call in filt_proc().

 kq = kn-kn_kq;
 KQ_LOCK(kq);
 if ((kn-kn_status  (KN_INFLUX | KN_SCAN)) == KN_INFLUX) {
   @@ -2385,15 +2385,16 @@ SYSINIT(knote, SI_SUB_PSEUDO, SI_ORDER_ANY, 
   knote_init, NULL);
static struct knote *
knote_alloc(int waitok)
{
   - return ((struct knote *)uma_zalloc(knote_zone,
   - (waitok ? M_WAITOK : M_NOWAIT)|M_ZERO));
   +
   + return (uma_zalloc(knote_zone, (waitok ? M_WAITOK : M_NOWAIT) |
   + M_ZERO));
}

static void
  
  per above, we should add inline here...
  
knote_free(struct knote *kn)
{
   - if (kn != NULL)
   - uma_zfree(knote_zone, kn);
   +
   + uma_zfree(knote_zone, kn);
}

/*
  
  I agree w/ the all the non-SLIST changes, but I disagree w/ the SLIST
  change as I don't believe that all cases was considered...
 What cases do you mean ?
 
 The patch does not unlock knlist lock in the iteration. As such, the
 only thread which could remove elements from the knlist, or rearrange
 the list, while loop is active, is the current thread. So I claim that
 the only the current iterating element can be removed, and the next list
 element stays valid. This is enough for _SAFE loop to work.

 Why do you think that _SAFE is incorrect ? Comment talks about very

I can't think of the reason right now, but I do remeber puzzling over
this issue for some hours when I wrote this code, and I had proved
to myself that _SAFE was NOT _SAFE for this use case...

In the quick look I just had, I have not been able to decide one way
or the other, but I'm suspicious that this is a recent issue, as this
code has been running for close to a decade w/o any issues, and wonder
if there was some other change that trigger the issue...

The reason I'm cautious about changing this is that the code has been
running fine for over a decade...  Have you done a full test to
validate that nothing else breaks?

Ok, after looking more at the original dump, this is a use after free
bug...  As I said in another email, it should not be possible to get
into the _FOREACH loop where knlist is an empty list.  If it does,
then there is another major bug that needs to be found...  A simple
change to _SAFE will not fix this use after free bug...

 different case, where the knlist lock is dropped. Then indeed, other
 thread may iterate in parallel, and invalidate the memoized next element
 while KN_INFLUX is set for the current element and knlist is dropped.
 But _SAFE in sys/queue.h never means 'safe

Re: Instant panic while trying run ports-mgmt/poudriere

2015-08-23 Thread John-Mark Gurney
/kern_event.c
 index a4388aa..2f15f7f 100644
 --- a/sys/kern/kern_event.c
 +++ b/sys/kern/kern_event.c
 @@ -1106,7 +1106,12 @@ kqueue_register(struct kqueue *kq, struct kevent *kev, 
 struct thread *td, int wa
   return EINVAL;
  
   if (kev-flags  EV_ADD)
 - tkn = knote_alloc(waitok);  /* prevent waiting with locks */
 + /*
 +  * Prevent waiting with locks.  Non-sleepable
 +  * allocation failures are handled in the loop, only
 +  * if the spare knote appears to be actually required.
 +  */
 + tkn = knote_alloc(waitok);

if you add this comment, please add curly braces around the block...

   else
   tkn = NULL;
  
 @@ -1310,8 +1315,7 @@ done:
   FILEDESC_XUNLOCK(td-td_proc-p_fd);
   if (fp != NULL)
   fdrop(fp, td);
 - if (tkn != NULL)
 - knote_free(tkn);
 + knote_free(tkn);

Probably should just change knote_free to a static inline that does
a uma_zfree as uma_zfree also does nothing is the input is NULL...

   if (fops != NULL)
   kqueue_fo_release(filt);
   return (error);
 @@ -1507,10 +1511,6 @@ kqueue_scan(struct kqueue *kq, int maxevents, struct 
 kevent_copyops *k_ops,
   } else
   asbt = 0;
   marker = knote_alloc(1);
 - if (marker == NULL) {
 - error = ENOMEM;
 - goto done_nl;
 - }
   marker-kn_status = KN_MARKER;
   KQ_LOCK(kq);
  
 @@ -1929,7 +1929,7 @@ void
  knote(struct knlist *list, long hint, int lockflags)
  {
   struct kqueue *kq;
 - struct knote *kn;
 + struct knote *kn, *tkn;
   int error;
  
   if (list == NULL)
 @@ -1948,7 +1948,7 @@ knote(struct knlist *list, long hint, int lockflags)
* only safe if you want to remove the current item, which we are
* not doing.
*/
 - SLIST_FOREACH(kn, list-kl_list, kn_selnext) {
 + SLIST_FOREACH_SAFE(kn, list-kl_list, kn_selnext, tkn) {

Clearly you didn't read the comment that preceeds this line, or at
least didn't update it:
 * SLIST_FOREACH, SLIST_FOREACH_SAFE is not safe in our case, it is
 * only safe if you want to remove the current item, which we are
 * not doing.

So, you'll need to be more specific in why this needs to change...
When I wrote this code, I spent a lot of time looking at this, and
reasoned as to why SLIST_FOREACH_SAFE was NOT correct usage here...

   kq = kn-kn_kq;
   KQ_LOCK(kq);
   if ((kn-kn_status  (KN_INFLUX | KN_SCAN)) == KN_INFLUX) {
 @@ -2385,15 +2385,16 @@ SYSINIT(knote, SI_SUB_PSEUDO, SI_ORDER_ANY, 
 knote_init, NULL);
  static struct knote *
  knote_alloc(int waitok)
  {
 - return ((struct knote *)uma_zalloc(knote_zone,
 - (waitok ? M_WAITOK : M_NOWAIT)|M_ZERO));
 +
 + return (uma_zalloc(knote_zone, (waitok ? M_WAITOK : M_NOWAIT) |
 + M_ZERO));
  }
  
  static void

per above, we should add inline here...

  knote_free(struct knote *kn)
  {
 - if (kn != NULL)
 - uma_zfree(knote_zone, kn);
 +
 + uma_zfree(knote_zone, kn);
  }
  
  /*

I agree w/ the all the non-SLIST changes, but I disagree w/ the SLIST
change as I don't believe that all cases was considered...

Anyways, the other changes shouldn't be committed w/ the SLIST change
as they are unrelated...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: bhyve: fix bhyve warning CTASSERT

2015-08-13 Thread John-Mark Gurney
Conrad Meyer wrote this message on Thu, Aug 13, 2015 at 08:12 -0700:
 Better to just replace CTASSERT() with _Static_assert() while you're here.

And make sure that sys/cdefs.h is included for compatibility w/ pre-C11
compilers...

 On Thu, Aug 13, 2015 at 5:05 AM, Stefano Garzarella
 stefanogarzare...@gmail.com wrote:
  Hi all,
  when I compile bhyve, I have the following errors from clang:
  pci_emul.c:750:2: error: unused typedef '__assert750'
  [-Werror,-Wunused-local-typedef]
  CTASSERT(sizeof(struct msicap) == 14);
  pci_emul.c:776:2: error: unused typedef '__assert776'
  [-Werror,-Wunused-local-typedef]
  CTASSERT(sizeof(struct msixcap) == 12);
  pci_emul.c:928:2: error: unused typedef '__assert928'
  [-Werror,-Wunused-local-typedef]
  CTASSERT(sizeof(struct pciecap) == 60);
 
  I fixed them in this simple way:
 
  diff --git a/bhyverun.h b/bhyverun.h
  index 87824ef..7ac3aa9 100644
  --- a/bhyverun.h
  +++ b/bhyverun.h
  @@ -32,7 +32,8 @@
   #ifndef CTASSERT   /* Allow lint to override */
   #defineCTASSERT(x) _CTASSERT(x, __LINE__)
   #define_CTASSERT(x, y) __CTASSERT(x, y)
  -#define__CTASSERT(x, y)typedef char __assert ## y[(x) ? 1
  : -1]
  +#define__CTASSERT(x, y)typedef char __assert ## y[(x) ? 1
  : -1] \
  +   __unused
   #endif

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Memory modified after free, seemingly geli related

2015-08-04 Thread John-Mark Gurney
Ed Maste wrote this message on Wed, Aug 05, 2015 at 03:24 +:
 I've encountered a few memory modified after free panics recently,
 which seem to be from geli. I don't yet have any debugging to
 completely confirm it's geli, but it has not happened on my other test
 laptop which configured similarly but without geli.

It is possible, but this doesn't tell us who last used the bio, just
that when geli was allocating a bio, that the newly allocated bio was
modified while it was free...

It's likely that r284861 is just exposed a previously existing bug in
the system...

You could try to use memguard(9) to help catch the modification when
it happens...

 This has a few local patches from my to-commit-to-HEAD queue.
 FreeBSD volta 11.0-CURRENT FreeBSD 11.0-CURRENT #10
 r284409+6a002d9(staging): Tue Jul  7 17:57:01 EDT 2015
 
 panic: Memory modified after free 0xf80009d504d8(248) val=0 @
 0xf80009d50518
 
 cpuid = 1
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe011414a880
 vpanic() at vpanic+0x189/frame 0xfe011414a900
 panic() at panic+0x43/frame 0xfe011414a960
 trash_ctor() at trash_ctor+0x48/frame 0xfe011414a970
 uma_zalloc_arg() at uma_zalloc_arg+0x573/frame 0xfe011414a9e0
 g_clone_bio() at g_clone_bio+0x1d/frame 0xfe011414aa00
 g_eli_start() at g_eli_start+0xbd/frame 0xfe011414aa30
 g_io_schedule_down() at g_io_schedule_down+0xe6/frame 0xfe011414aa60
 g_down_procbody() at g_down_procbody+0x7d/frame 0xfe011414aa70
 fork_exit() at fork_exit+0x84/frame 0xfe011414aab0
 fork_trampoline() at fork_trampoline+0xe/frame 0xfe011414aab0
 --- trap 0, rip = 0, rsp = 0xfe011414ab70, rbp = 0 ---
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: IPSEC stop works after r285336

2015-08-03 Thread John-Mark Gurney
Sydney Meyer wrote this message on Mon, Aug 03, 2015 at 01:15 +0200:
 the revision i built included gnn's patches to setkey already.
 
 I have tried to setup a tunnel using strongswan with gcm as esp cipher mode, 
 but the connection fails with algorithm AES_GCM_16 not supported by kernel..

It looks like GCM isn't compiled by default by the port...  Try the
attached patch to
src/libhydra/plugings/kernel_pfkey/kernel_pfkey_ipsec.c...  it may
require more modifications...

Someone else would be better to work on this...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
--- kernel_pfkey_ipsec.c.orig	2015-08-03 17:15:48.676749000 -0700
+++ kernel_pfkey_ipsec.c	2015-08-03 17:16:40.987182000 -0700
@@ -822,13 +822,13 @@
 /*	{ENCR_DES_IV32,0			}, */
 	{ENCR_NULL,	SADB_EALG_NULL},
 	{ENCR_AES_CBC,SADB_X_EALG_AESCBC			},
-/*	{ENCR_AES_CTR,SADB_X_EALG_AESCTR			}, */
+	{ENCR_AES_CTR,SADB_X_EALG_AESCTR			},
 /*  {ENCR_AES_CCM_ICV8,			SADB_X_EALG_AES_CCM_ICV8	}, */
 /*	{ENCR_AES_CCM_ICV12,		SADB_X_EALG_AES_CCM_ICV12	}, */
 /*	{ENCR_AES_CCM_ICV16,		SADB_X_EALG_AES_CCM_ICV16	}, */
 /*	{ENCR_AES_GCM_ICV8,			SADB_X_EALG_AES_GCM_ICV8	}, */
 /*	{ENCR_AES_GCM_ICV12,		SADB_X_EALG_AES_GCM_ICV12	}, */
-/*	{ENCR_AES_GCM_ICV16,		SADB_X_EALG_AES_GCM_ICV16	}, */
+	{ENCR_AES_GCM_ICV16,		SADB_X_EALG_AESGCM16	},
 	{END_OF_LIST,0			},
 };
 
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: IPSEC stop works after r285336

2015-08-01 Thread John-Mark Gurney
Sydney Meyer wrote this message on Sun, Aug 02, 2015 at 04:03 +0200:
 i have tried your patches from your ipsecgcm branch. The build completes, 
 boots fine and indeed, dmesg shows aesni0: AES-CBC,AES-XTS,AES-GCM,AES-ICM 
 on motherboard.

Yeh, these patches are more about getting IPsec to work w/ the modes
that aesni now supports...

 I'm going to try out the new cipher modes tomorrow and will get back..

Make sure you get the gnn's setkey changes in r286143 otherwise GCM
and CTR won't work...

Thanks for doing more testing.. I've only done basic ping tests, so
passing more real traffic through would be nice...

  On 01 Aug 2015, at 22:01, John-Mark Gurney j...@funkthat.com wrote:
  
  Sydney Meyer wrote this message on Wed, Jul 29, 2015 at 22:01 +0200:
  Same here, fixed running r286015. Thanks a  bunch.
  
  If you'd like to do some more testing, test the patches in:
  https://github.com/jmgurney/freebsd/tree/ipsecgcm
  
  These patches get GCM and CTR modes working as tested against NetBSD
  6.1.5...
  
  Hope to commit these in the next few days..
  
  Thanks.
  
  On 29 Jul 2015, at 14:56, Alexandr Krivulya shur...@shurik.kiev.ua 
  wrote:
  
  29.07.2015 10:17, John-Mark Gurney ??:
  Alexandr Krivulya wrote this message on Thu, Jul 23, 2015 at 10:38 +0300:
  
  [...]
  
  With r285535 all works fine.
  Sydney Meyer wrote this message on Mon, Jul 27, 2015 at 23:49 +0200:
  I'm having the same problem with IPSec, running -current with r285794.
  
  Don't know if this helps, but netstat -s -p esp shows packets 
  dropped; bad ilen.
  It looks like there was an issue w/ that commit...  After looking at
  the code, and working w/ gnn, I have committed r286000 which fixes it
  in my test cases...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: IPSEC stop works after r285336

2015-08-01 Thread John-Mark Gurney
Sydney Meyer wrote this message on Wed, Jul 29, 2015 at 22:01 +0200:
 Same here, fixed running r286015. Thanks a  bunch.

If you'd like to do some more testing, test the patches in:
https://github.com/jmgurney/freebsd/tree/ipsecgcm

These patches get GCM and CTR modes working as tested against NetBSD
6.1.5...

Hope to commit these in the next few days..

Thanks.

  On 29 Jul 2015, at 14:56, Alexandr Krivulya shur...@shurik.kiev.ua wrote:
  
  29.07.2015 10:17, John-Mark Gurney ??:
  Alexandr Krivulya wrote this message on Thu, Jul 23, 2015 at 10:38 +0300:
  
  [...]
  
  With r285535 all works fine.
  Sydney Meyer wrote this message on Mon, Jul 27, 2015 at 23:49 +0200:
  I'm having the same problem with IPSec, running -current with r285794.
  
  Don't know if this helps, but netstat -s -p esp shows packets dropped; 
  bad ilen.
  It looks like there was an issue w/ that commit...  After looking at
  the code, and working w/ gnn, I have committed r286000 which fixes it
  in my test cases...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: FreeBSD_HEAD_i386 - Build #721 - Failure

2015-07-30 Thread John-Mark Gurney
 not permitted
 rm: FreeBSD_HEAD_i386/usr/bin/ypchpass: Operation not permitted
 rm: FreeBSD_HEAD_i386/usr/bin/login: Operation not permitted
 rm: FreeBSD_HEAD_i386/usr/bin: Directory not empty
 rm: FreeBSD_HEAD_i386/usr/lib/librt.so.1: Operation not permitted
 rm: FreeBSD_HEAD_i386/usr/lib: Directory not empty
 rm: FreeBSD_HEAD_i386/usr: Directory not empty
 rm: FreeBSD_HEAD_i386/sbin/init: Operation not permitted
 rm: FreeBSD_HEAD_i386/sbin: Directory not empty
 rm: FreeBSD_HEAD_i386/lib/libthr.so.3: Operation not permitted
 rm: FreeBSD_HEAD_i386/lib/libcrypt.so.5: Operation not permitted
 rm: FreeBSD_HEAD_i386/lib/libc.so.7: Operation not permitted
 rm: FreeBSD_HEAD_i386/lib: Directory not empty
 rm: FreeBSD_HEAD_i386: Directory not empty
 + true
 + sudo chflags -R noschg FreeBSD_HEAD_i386
 + sudo rm -fr FreeBSD_HEAD_i386
 Email was triggered for: Failure - Any
 Sending email for trigger: Failure - Any

I believe a simple #include sys/systm.h in sys/net/pfkeyv2.h is all
that is needed... I'm testing this fix now...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: r285947: broken AESNI support? No aesni0 on Intel XEON E5-1650-v3 on Fujitsu Celsius M740

2015-07-29 Thread John-Mark Gurney
O. Hartmann wrote this message on Wed, Jul 29, 2015 at 10:20 +0200:
 On Wed, 29 Jul 2015 00:36:16 -0700
 John-Mark Gurney j...@funkthat.com wrote:
 
  O. Hartmann wrote this message on Wed, Jul 29, 2015 at 07:39 +0200:
   Running a workstation with CURRENT (FreeBSD 11.0-CURRENT #5 r285947: Tue
   Jul 28 13:39:03 CEST 2015 amd64) equipted with an Intel XEON E5-1650 v3,
   see the extraction from recent dmesg below.
   
   I double checked the UEFI settings (the box is a Fujitsu Celsius M740 with
   most recent firmware 1.8.0) and I didn't find anything indicating that
   AES-NI has been deactivated.
   
   I checked the data sheet at Intel, the CPU should support AES-NI.
   
   I also filed a PR: Bug 201960 
   
   I'd like to know whether this is by intention, by bug (feature mask 
   wrong?)
   or by a faulty firmware? Any hints?
  
  Can you send me the output of cpuid-etallen?  It's pretty long, so
  maybe off list would be better...  It's from a port of the same
  name...
 
 I'm sorry, since I work in a pretty restricted area, I can not offer
 webspace-similar download areas, but if it is not offending the list, i could
 provide a compressed output.
 
 Find the output attached xz-compressed ... I cleared intentionally the serial
 number, just as a notice.

Yep, this confirms that AES-NI is off:
  AES instruction = false

Which isn't a surprise from our other data points.  Just wanted to
make sure...

  Also, it looks like a microcode update could fix this issue, have you
  tried to look at that?
  
  https://albertveli.wordpress.com/2013/03/05/aes-ni-enabled/
  
  Looks very similar to your issue, though it's a different microarch..
  Your's is a Haswell that has the TSX bug in it, and it could be that
  the bios is disabling too many feature bits...
  
  Have you made sure that your machine has the latest BIOS?  A newer
  BIOS could reenable the feature too...
 
 I just checked this moment again, but the latest UEFI firmware Fujitsu is
 offering is version 1.8.0 from April of this year.

I would complain to the vendor of your machine...  I'd contact them
and try to return the machine as defective...  It clearly is...

   [...]
   FreeBSD clang version 3.6.1 (tags/RELEASE_361/final 237755) 20150525
   VT: running with driver efifb.
   CPU: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz (3491.98-MHz K8-class CPU)
 Origin=GenuineIntel  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
 
   Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE
 
   Features2=0x7dfefbffSSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,XSAVE,OSXSAVE,AVX,F16C,RDRAND
  
  There should be an AESNI feature on this line, but clearly not...
 
 On another machine, also Fujitsu, but a 19 inch rack server module with a low
 energy XEON E5-12XXv3, I can clearly see the AESNI feature in Feature2 list
 and, conclusively, the aesni0 device is present and reported.
 
  
  [...]
  
   aesni0: No AESNI support.
   [...]
  
  Which is why you get this...
  
 
 I applied the port sysutils/cpuid to another system, runnin a i5-4200M mobile
 CPU (Lenovo notebook). The rows indicating
 
 family  = Intel ...
 (simple synth)  = ...
 
 look much more modern for my opinion as the output I provided shows on the
 CPU in question. It is just a hunch ... Seems, I've bought Intel(ian) crap ;-)
 with no features and from another mellenium ...

Yep...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: r285947: broken AESNI support? No aesni0 on Intel XEON E5-1650-v3 on Fujitsu Celsius M740

2015-07-29 Thread John-Mark Gurney
O. Hartmann wrote this message on Wed, Jul 29, 2015 at 07:39 +0200:
 Running a workstation with CURRENT (FreeBSD 11.0-CURRENT #5 r285947: Tue Jul 
 28
 13:39:03 CEST 2015 amd64) equipted with an Intel XEON E5-1650 v3, see the
 extraction from recent dmesg below.
 
 I double checked the UEFI settings (the box is a Fujitsu Celsius M740 with 
 most
 recent firmware 1.8.0) and I didn't find anything indicating that AES-NI has
 been deactivated.
 
 I checked the data sheet at Intel, the CPU should support AES-NI.
 
 I also filed a PR: Bug 201960 
 
 I'd like to know whether this is by intention, by bug (feature mask wrong?) or
 by a faulty firmware? Any hints?

Can you send me the output of cpuid-etallen?  It's pretty long, so
maybe off list would be better...  It's from a port of the same
name...

Also, it looks like a microcode update could fix this issue, have you
tried to look at that?

https://albertveli.wordpress.com/2013/03/05/aes-ni-enabled/

Looks very similar to your issue, though it's a different microarch..
Your's is a Haswell that has the TSX bug in it, and it could be that
the bios is disabling too many feature bits...

Have you made sure that your machine has the latest BIOS?  A newer
BIOS could reenable the feature too...

 [...]
 FreeBSD clang version 3.6.1 (tags/RELEASE_361/final 237755) 20150525
 VT: running with driver efifb.
 CPU: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz (3491.98-MHz K8-class CPU)
   Origin=GenuineIntel  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
   
 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE
   
 Features2=0x7dfefbffSSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,XSAVE,OSXSAVE,AVX,F16C,RDRAND

There should be an AESNI feature on this line, but clearly not...

[...]

 aesni0: No AESNI support.
 [...]

Which is why you get this...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: IPSEC stop works after r285336

2015-07-29 Thread John-Mark Gurney
Alexandr Krivulya wrote this message on Thu, Jul 23, 2015 at 10:38 +0300:

[...]

 With r285535 all works fine.

Sydney Meyer wrote this message on Mon, Jul 27, 2015 at 23:49 +0200:
 
 I'm having the same problem with IPSec, running -current with r285794.
 
 Don't know if this helps, but netstat -s -p esp shows packets dropped; bad 
 ilen.

It looks like there was an issue w/ that commit...  After looking at
the code, and working w/ gnn, I have committed r286000 which fixes it
in my test cases...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: IPSEC stop works after r285336

2015-07-24 Thread John-Mark Gurney
Alexandr Krivulya wrote this message on Thu, Jul 23, 2015 at 10:38 +0300:
 I have IPSEC tunnel inside l2tp tunnel via mpd. After r285536 I see only
 outgoing esp packets on ng interface:

This change is -stable, not -current, but the change referenced below
is -current... Which one are you running?

Also, the only ipsec related change after r285535 is r285770, though
that probably won't effect it...  Could you possibly narrow the change
that broke things?

 root@thinkpad:/usr/src # tcpdump -i ng0
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on ng0, link-type NULL (BSD loopback), capture size 262144 bytes
 10:35:27.331886 IP 10.10.10.2  10.10.10.1:
 ESP(spi=0x03081e58,seq=0x9a5), length 140
 10:35:28.371707 IP 10.10.10.2  10.10.10.1:
 ESP(spi=0x03081e58,seq=0x9a6), length 140
 10:35:29.443536 IP 10.10.10.2  10.10.10.1:
 ESP(spi=0x03081e58,seq=0x9a7), length 140
 10:35:30.457370 IP 10.10.10.2  10.10.10.1:
 ESP(spi=0x03081e58,seq=0x9a8), length 140
 10:35:31.475606 IP 10.10.10.2  10.10.10.1:
 ESP(spi=0x03081e58,seq=0x9a9), length 140
 10:35:31.622315 IP 10.10.10.1.isakmp  10.10.10.2.isakmp: isakmp: phase
 2/others ? inf[E]
 10:35:31.622544 IP 10.10.10.2.isakmp  10.10.10.1.isakmp: isakmp: phase
 2/others ? inf[E]
 10:35:31.622658 IP 10.10.10.2.isakmp  10.10.10.1.isakmp: isakmp: phase
 2/others ? inf[E]
 10:35:31.623933 IP 10.10.10.1.isakmp  10.10.10.2.isakmp: isakmp: phase
 2/others ? inf[E]
 10:35:32.492349 IP 10.10.10.2  10.10.10.1:
 ESP(spi=0x03081e58,seq=0x9aa), length 140
 10:35:33.509346 IP 10.10.10.2  10.10.10.1:
 ESP(spi=0x03081e58,seq=0x9ab), length 140
 10:35:34.527187 IP 10.10.10.2  10.10.10.1:
 ESP(spi=0x03081e58,seq=0x9ac), length 140
 10:35:35.539600 IP 10.10.10.2  10.10.10.1:
 ESP(spi=0x03081e58,seq=0x9ad), length 140
 
 With r285535 all works fine.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Instant panic while trying run ports-mgmt/poudriere

2015-07-15 Thread John-Mark Gurney
Pawel Pekala wrote this message on Wed, Jul 15, 2015 at 17:46 +0200:
 On 2015-07-14 15:38 -0700, John-Mark Gurney j...@funkthat.com wrote:
 Pawel Pekala wrote this message on Mon, Jul 13, 2015 at 23:12 +0200:
  Let me know if you need more details.
 
 Were you running X at the time of the crash?  and if so, can you try
 to reproduce w/o X running?  It's hard to know if the panic (and you
 didn't include the panic string) is due to kern_event, or trying to
 do too much in the console driver.
 
 Thanks.
 
 Last tests were done with X running yes. Today I did same test with all
 services commented out in rc.conf (including X) and did get same result.
 Poudriere causes kernel panic always in the same spot:
 
 [00:00:39]  Calculating ports order and dependencies

Please repost the entire panic message, and the back trace w/o X
running...  Also, if you could share the core and kernel w/ me (you can
email me directly if you'd like), that'd help.

Thanks.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Instant panic while trying run ports-mgmt/poudriere

2015-07-14 Thread John-Mark Gurney
Pawel Pekala wrote this message on Mon, Jul 13, 2015 at 23:12 +0200:
 Let me know if you need more details.

Were you running X at the time of the crash?  and if so, can you try
to reproduce w/o X running?  It's hard to know if the panic (and you
didn't include the panic string) is due to kern_event, or trying to
do too much in the console driver.

Thanks.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Instant panic while trying run ports-mgmt/poudriere

2015-07-14 Thread John-Mark Gurney
Pawel Pekala wrote this message on Tue, Jul 14, 2015 at 22:47 +0200:
 On 2015-07-13 23:28 +0200, Mateusz Guzik mjgu...@gmail.com wrote:
 On Mon, Jul 13, 2015 at 11:12:05PM +0200, Pawel Pekala wrote:
  Hi
  
  I'm getting 100% reproducible kernel crash while trying build ports
  with poudriere on my system. This started to show up about 2-3 weeks
  ago. I upgrade my system on weekly basis usually on saturday.
  Here's backtrace:
  
  (kgdb) bt
 [..]
  at /hdd/src/sys/amd64/amd64/trap.c:201
  #25 0x80dace32 in calltrap ()
  at /hdd/src/sys/amd64/amd64/exception.S:235 #26 0x80941430
  in knote (list=0xf801a2589408, hint=2147483648, lockflags=value
  optimized out) at /hdd/src/sys/kern/kern_event.c:1920 #27
  0x80946a51 in exit1 (td=0xf801b84014d0, rv=value
  optimized out) at /hdd/src/sys/kern/kern_exit.c:560 #28
  0x80945f1e in sys_sys_exit (td=0x0, uap=value optimized
  out) at /hdd/src/sys/kern/kern_exit.c:178 #29 0x80dcdaa2 in
  outamd64_syscall (td=0xf801b84014d0, traced=0)
  at subr_syscall.c:133
  #30 0x80dad11b in Xfast_syscall ()
  at /hdd/src/sys/amd64/amd64/exception.S:395 #31 0x000800922eea
  in ?? () Previous frame inner to this frame (corrupt stack?)
  Current language:  auto; currently minimal
  
  Let me know if you need more details.
 
 
 Well, if the problem is really that reproducible it would be best if
 you narrowed it down to the exact commit.
 
 However, quick look suggests you may be a victim of r284861.
 
 After further testing I can confirm that this panic was introduced in
 r284861, thanks for the hint!

Can you tell me what your line 1920 of kern_event.c is? (and the context
around it?   Or at least the $FreeBSD$ line from kern_event.c?  Because
in HEAD, the line is:
} else if ((lockflags  KNF_NOKQLOCK) != 0) {

and there isn't a way to fault on that code...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Failed to build 11-CURRENT on 10.1-RELEASE (was: Re: FreeBSD_HEAD - Build #2953 - Still Failing

2015-07-12 Thread John-Mark Gurney
Li-Wen Hsu wrote this message on Sun, Jul 12, 2015 at 11:31 +0800:
 Resend this using another subject for not being overwhelmed by lots of build 
 fail mails :(

This has already been addressed by r285417:
svnweb.freebsd.org/changeset/base/r285417

 On Sun, Jul 12, 2015 at 01:22:48 +0800, Li-Wen Hsu wrote:
  On Sat, Jul 11, 2015 at 17:11:24 +, jenkins-ad...@freebsd.org wrote:
   FreeBSD_HEAD - Build #2953 - Still Failing:
   
   Build information: https://jenkins.FreeBSD.org/job/FreeBSD_HEAD/2953/
   Full change log: https://jenkins.FreeBSD.org/job/FreeBSD_HEAD/2953/changes
   Full build log: https://jenkins.FreeBSD.org/job/FreeBSD_HEAD/2953/console
  
  [...]
  
   The end of the build log:
  
  [...]
   
   cc -O2 -pipe  -I/builds/FreeBSD_HEAD/usr.bin/xinstall/../../contrib/mtree 
   -I/builds/FreeBSD_HEAD/usr.bin/xinstall/../../lib/libnetbsd 
   -I/builds/FreeBSD_HEAD/usr.bin/xinstall/../../lib/libmd -std=gnu99  
   -Qunused-arguments 
   -I/builds/FreeBSD_HEAD/obj/builds/FreeBSD_HEAD/tmp/legacy/usr/include  
   -static -L/builds/FreeBSD_HEAD/obj/builds/FreeBSD_HEAD/tmp/legacy/usr/lib 
   -o xinstall xinstall.o getid.o  -lmd -legacy
   xinstall.o: In function `install':
   /builds/FreeBSD_HEAD/usr.bin/xinstall/xinstall.c:(.text+0x133c): 
   undefined reference to `_libmd_SHA256_File'
   /builds/FreeBSD_HEAD/usr.bin/xinstall/xinstall.c:(.text+0x1631): 
   undefined reference to `_libmd_SHA256_End'
   /builds/FreeBSD_HEAD/usr.bin/xinstall/xinstall.c:(.text+0x186c): 
   undefined reference to `_libmd_SHA256_File'
   xinstall.o: In function `compare':
   /builds/FreeBSD_HEAD/usr.bin/xinstall/xinstall.c:(.text+0x2706): 
   undefined reference to `_libmd_SHA256_End'
   cc: error: linker command failed with exit code 1 (use -v to see 
   invocation)
   *** [xinstall] Error code 1
   
   make[3]: stopped in /builds/FreeBSD_HEAD/usr.bin/xinstall
   1 error
   
   make[3]: stopped in /builds/FreeBSD_HEAD/usr.bin/xinstall
  
  This failure seems due to cannot build 11-CURRENT on 10.1-RELEASE, where
  this job configured to run on.
  
  I have tested building HEAD in 11-CURRENT and 10.1-RELEASE jails,
  the first one succeeded and the other failed.  
  
  I remember we need to support build -CURRENT on latest -RELEASE, right?
  Any comments?

Please keep up w/ commit mail in situations like these...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Security issue when using aesni(4) module with only ESP on HEAD

2015-07-06 Thread John-Mark Gurney
It has been discovered that r275732[1] on HEAD introduced a bug in the
aesni(4) module where the initialization vector (IV) is not properly
generated when using AES-CBC, aka rijndael-cbc.  This only happens
when both the CRD_F_IV_PRESENT and CRD_F_IV_EXPLICIT flags are not
set.  This ONLY affects HEAD and does not affect any stable branch
as the code in r275732 has not yet been back ported.

The only happen when the system is running IPsec and has a security
policy that only includes encryption (ESP).  If an authentication
policy (AH) is specified along with an encryption policy, which is the
recommended configuration to prevent an attacker from modifying packets,
the aesni(4) module will not be used, and this bug will not affect the
policy.

This bug has been fixed in r285216[2].  Please upgrade immediately if you
are using IPsec w/ an ESP only policy and the aesni(4) module.

The bug will leak the XOR difference[3] of the first 16 bytes of the
packet, and possibly more.  In tunnel mode, this only covers part of
the IP header, including the internal source IP.  In transport mode,
most of the TCP header will be leaked and the header and first 8 bytes 
of a UDP packet.

Other subsystems in FreeBSD, kgssapi, geli and cryptodev, set the
CRD_F_IV_PRESENT and/or CRD_F_IV_EXPLICIT flags and are not affected
by this bug.

Thanks go to Olivier Cochard-Labbé for reporting a related
issue and discovering that the packet IVs were not properly random.

[1] https://svnweb.freebsd.org/changeset/base/r275732
[2] https://svnweb.freebsd.org/changeset/base/r285216
[3] https://defuse.ca/cbcmodeiv.htm

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: How should a driver shutdown a taskqueue on detach?

2015-07-01 Thread John-Mark Gurney
Ryan Stone wrote this message on Wed, Jul 01, 2015 at 15:44 -0400:
 I'm trying to figure out how a driver is supposed to shut down its
 interrupt-handling taskqueue when it detaches.  taskqueue(9) recommends
 disabling interrupts, draining each task and then freeing the taskqueue.
 The problem that I have is the interrupt-handling tasks will sometimes
 re-enable interrupts on the device.  Is there a better way than using some
 kind of flag internally in the driver to note that a detach is in progress
 that the interrupt handlers will have to check before enabling interrupts?

Why not disabled interrupts, unregister interrupt handler, and then
make sure interrupts are disabled (only needed to prevent an interrupt
storm)?  Once you have unregistered the interrupt handler, it can't run
again...  Then you're free to drain the task queue safely...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: proper way or unworkable idea ?

2015-06-30 Thread John-Mark Gurney
Jeffrey Bouquet wrote this message on Mon, Jun 29, 2015 at 04:10 -0700:
 If I've a spare /mnt/usr/src , it seems buildworld quite
 soon fails, where it otherwise may succeed in /usr/src. Any CLI parameters or 
 the build system is hardcoded enough so that there will always be problems? 

Doing a buildworld from a source tree not located in /usr/src works
and I do it all the time...  There is nothing special, simple
buildworld just works...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Increase BUFSIZ to 8192

2015-05-14 Thread John-Mark Gurney
Ian Lepore wrote this message on Wed, May 13, 2015 at 12:47 -0600:
 On Wed, 2015-05-13 at 11:13 -0700, John-Mark Gurney wrote:
  Adrian Chadd wrote this message on Wed, May 13, 2015 at 08:34 -0700:
   The reason I ask about why is it faster? is because for embedded-y
   things with low RAM we may not want that to happen due to memory
   constraints. However, we may actually want to do some form of
   autotuning on some platforms.
  
  If you're already running a program, the difference between 1k and
  8k isn't significant... I'll give you 64k can be significant for
  embedded-y platforms...  But this goes back to the, we need a global
  knob saying I want low memory usage, and I am willing to pay for it
  in performance...
 
 It is NOT just a difference of 1K vs 8K.  It's that much times however
 many BUFSIZ-sized things a program has allocated at once.  It's where
 they are allocated.  As I've already pointed out, BUFSIZ appears in the
 base code over 2000 times.  Where is the analysis of the impact an 8x
 change is going to have on all those uses?

Since you apprently missed my original reply, I said that we shouldn't
abuse BUFSIZ for this work, and that it should be changed in mdXhl.c...

I agree that changing this size to effect all the other files is ill
advised and should not be done...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Increase BUFSIZ to 8192

2015-05-14 Thread John-Mark Gurney
David Chisnall wrote this message on Wed, May 13, 2015 at 09:27 +0100:
 On 13 May 2015, at 09:03, John-Mark Gurney j...@funkthat.com wrote:
  
  Poul-Henning Kamp wrote this message on Tue, May 12, 2015 at 06:31 +:
  
  In message 20150512032307.gp37...@funkthat.com, John-Mark Gurney writes:
  
  Also, you'd probably see even better performance by increasing the
  size to 64k, [...]
  
  easy:
 8K on 32bit
 64k on 64bit
  
  Sounds good to me...  Just for people who care... I did a quick set of
  benchmarks on sha256.. This is using my preliminary patch to use sse4
  optimized sha256...  But this should be the same for others...
  
  The numbers in ministat output are the time in seconds it takes my
  3.4GHz AMD A10-5700 APU running HEAD to process a 512MB file, so lower
  numbers are better..  I've processed them into easier to read format:
  BUFSIZ: 145MB/sec
  8k: 193MB/sec
  16k:198MB/sec
  64k:202MB/sec
  128k:   202MB/sec
  -t: 211MB/sec
 
 It looks like most of the benefit is gained at 16KB.  Did you try running the 
 benchmark with something else running at the same time to see if there is any 
 advantage in trashing the caches a bit less (simple case, what happens if you 
 run two instances of the same benchmark at once)?
 
 I suspect that you???re about right anyway - I recently did some tests while 
 playing with JavaScript FFI generation with a multithreaded process 
 JavaScript environment calling out to OpenSSL to do SHA calculations and 
 having each of 8 threads reading in 128KB chunks gave the fastest performance 
 (Core i7, 4 cores + hyperthreading), with only a negligible gain over 64KB.  
 In all cases, the JavaScript implementation was significantly faster than the 
 openssl tool, which used 8KB buffers.

Just in case anyone else wants to know how to run benchmarks
themselves..  Go into /usr/src/lib/libmd, edit mdXhl.c, and change
the occurence of BUFSIZ to what you want to test, say 64*1024, run:
make all  make install

and then you can run programs like sha256 -t, or:
for i in `jot 5 1`; do /usr/bin/time sha256 test.file ; done 2 XXX.times

Where test.file is populated maybe like:
dd if=/dev/urandom of=test.file bs=1m count=512

Then run:
ministat XXX.times YYY.times

to compare multiple results...

Happy benchmarking!

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Increase BUFSIZ to 8192

2015-05-14 Thread John-Mark Gurney
Poul-Henning Kamp wrote this message on Thu, May 14, 2015 at 07:42 +:
 
 In message 20150514072155.gt37...@funkthat.com, John-Mark Gurney writes:
 
 Since you apprently missed my original reply, I said that we shouldn't
 abuse BUFSIZ for this work, and that it should be changed in mdXhl.c...
 
 Say what ?
 
 BUFSIZ is used entirely appropriately in MDXFileChunk():  For reading
 a file into an algorithm.

Posix-2008:
BUFSIZ: Size of stdio.h buffers.  This shall expand to a positive value.

C99:
BUFSIZ
which expands to an integer constant expression that is the size of
the buffer used by the setbuf function;

In fact, posix-2008 references LINE_MAX because:
Frequently, utility writers selected the UNIX system constant BUFSIZ to
allocate these buffers; therefore, some utilities were limited to 512
bytes for I/O lines, while others achieved 4 096 bytes or greater.

BUFSIZ was already recognized as to small to hold a single line, yet
you're saying it's perfectly fine to use as a buffer for binary data?

 If in stead of open(2), fopen(3) had been used, the exact same thing
 would happen, but using malloc space rather than stack space.

Plus extra overhead.. :)

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Increase BUFSIZ to 8192

2015-05-13 Thread John-Mark Gurney
Hans Petter Selasky wrote this message on Wed, May 13, 2015 at 10:35 +0200:
 On 05/13/15 10:27, David Chisnall wrote:
  On 13 May 2015, at 09:03, John-Mark Gurney j...@funkthat.com wrote:
 
  Poul-Henning Kamp wrote this message on Tue, May 12, 2015 at 06:31 +:
  
  In message 20150512032307.gp37...@funkthat.com, John-Mark Gurney writes:
 
  Also, you'd probably see even better performance by increasing the
  size to 64k, [...]
 
  easy:
8K on 32bit
64k on 64bit
 
  Sounds good to me...  Just for people who care... I did a quick set of
  benchmarks on sha256.. This is using my preliminary patch to use sse4
  optimized sha256...  But this should be the same for others...
 
  The numbers in ministat output are the time in seconds it takes my
  3.4GHz AMD A10-5700 APU running HEAD to process a 512MB file, so lower
  numbers are better..  I've processed them into easier to read format:
  BUFSIZ:145MB/sec
  8k:193MB/sec
  16k:   198MB/sec
  64k:   202MB/sec
  128k:  202MB/sec
  -t:211MB/sec
 
  It looks like most of the benefit is gained at 16KB.  Did you try running 
  the benchmark with something else running at the same time to see if there 
  is any advantage in trashing the caches a bit less (simple case, what 
  happens if you run two instances of the same benchmark at once)?
 
  I suspect that you???re about right anyway - I recently did some tests 
  while playing with JavaScript FFI generation with a multithreaded process 
  JavaScript environment calling out to OpenSSL to do SHA calculations and 
  having each of 8 threads reading in 128KB chunks gave the fastest 
  performance (Core i7, 4 cores + hyperthreading), with only a negligible 
  gain over 64KB.  In all cases, the JavaScript implementation was 
  significantly faster than the openssl tool, which used 8KB buffers.
 
 You should also try this using an USB disk. The performance numbers 
 heavily depends on the hardware's interrupt moderation values.

This shouldn't matter.. I wasn't flushing the buffer cache between
runs, so this was entirely from the buffer cache...  This is purely,
syscall+copy overhead that is being measured here...  No matter what
you're source is, NFS, USB disk, you'll always have this overhead...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Increase BUFSIZ to 8192

2015-05-13 Thread John-Mark Gurney
Adrian Chadd wrote this message on Wed, May 13, 2015 at 08:34 -0700:
 The reason I ask about why is it faster? is because for embedded-y
 things with low RAM we may not want that to happen due to memory
 constraints. However, we may actually want to do some form of
 autotuning on some platforms.

If you're already running a program, the difference between 1k and
8k isn't significant... I'll give you 64k can be significant for
embedded-y platforms...  But this goes back to the, we need a global
knob saying I want low memory usage, and I am willing to pay for it
in performance...

 So, if it's underlying block size, maybe BUFSIZ isn't the thing to
 tweak, but based on disk io buffer size.
 If it's filling L1 or L2 cache with useful work, maybe auto-tune it
 based on that.

I'm pretty sure this is just simply, syscalls+copies are expensive,
and larger block sizes reduces the number of calls, going from 1k to
64k means 64 times less syscalls...

So, in my benchmark, we went from 148271 syscalls/second to 3228
syscalls/second for 64k block size, and we got a 40% perf increase on
top of this...  i.e. we spend ~40% of the cpu time to do 145k syscalls
instead of doing real work...

 Please don't take this as bikeshedding, I'd really like to see some
 this is why it's faster analysis rather than just numbers thrown
 around.

I don't really see a need to analyize this any more... We are batching
work in a more effecient manner...  I could list many other examples
of where we do similar optimizations...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Increase BUFSIZ to 8192

2015-05-13 Thread John-Mark Gurney
Poul-Henning Kamp wrote this message on Tue, May 12, 2015 at 06:31 +:
 
 In message 20150512032307.gp37...@funkthat.com, John-Mark Gurney writes:
 
 Also, you'd probably see even better performance by increasing the
 size to 64k, [...]
 
 easy:
   8K on 32bit
   64k on 64bit

Sounds good to me...  Just for people who care... I did a quick set of
benchmarks on sha256.. This is using my preliminary patch to use sse4
optimized sha256...  But this should be the same for others...

The numbers in ministat output are the time in seconds it takes my
3.4GHz AMD A10-5700 APU running HEAD to process a 512MB file, so lower
numbers are better..  I've processed them into easier to read format:
BUFSIZ: 145MB/sec
8k: 193MB/sec
16k:198MB/sec
64k:202MB/sec
128k:   202MB/sec
-t: 211MB/sec

x def.times
+ 8k.times
* 16k.times
% 64k.times
# 128k.times
+-+
|#%  *+ x |
|#%  *+ x |
|#%  *+ x |
|##  *+ xx|
|A|  AA|A||
+-+
N   Min   MaxMedian   AvgStddev
x   5  3.53  3.55  3.53 3.536  0.0089442719
+   5  2.65  2.66  2.65 2.654  0.0054772256
Difference at 95.0% confidence
-0.882 +/- 0.0108161
-24.9434% +/- 0.305885%
(Student's t, pooled s = 0.0074162)
*   5  2.58  2.59  2.58 2.584  0.0054772256
Difference at 95.0% confidence
-0.952 +/- 0.0108161
-26.9231% +/- 0.305885%
(Student's t, pooled s = 0.0074162)
%   5  2.53  2.54  2.54 2.538   0.004472136
Difference at 95.0% confidence
-0.998 +/- 0.0103127
-28.224% +/- 0.29165%
(Student's t, pooled s = 0.00707107)
#   5  2.53  2.54  2.53 2.532   0.004472136
Difference at 95.0% confidence
-1.004 +/- 0.0103127
-28.3937% +/- 0.29165%
(Student's t, pooled s = 0.00707107)

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Increase BUFSIZ to 8192

2015-05-11 Thread John-Mark Gurney
Baptiste Daroussin wrote this message on Tue, May 12, 2015 at 01:06 +0200:
 I would like to change the default value of BUFSIZ to 8192.
 
 After testing various applications that uses BUFSIZ like changing it in libmd 
 I
 can often see performance improvements:
 
 For example with md5(1):
 Current BUFSIZ (1024)
 0.13 real 0.04 user 0.09 sys
 New BUFSIZ (8192)
 0.08 real 0.04 user 0.03 sys
 
 sha256(1):
 Before:
 0.44 real 0.39 user 0.04 sys
 After:
 0.37 real 0.35 user 0.01 sys
 
 This is done on a small amd64 Lenovo S20 laptop
 
 Review available here:
 https://reviews.freebsd.org/D2515

personally, I think the applications that are abusing BUFSIZ should be
fixed to use a properly sized buffer for their applications...  BUFSIZ
is defined for the default stdio buffer sizes...

I got significant perf improvement many years ago by fixed lpd to use
a sane BUFSIZ...  And did the same recently w/ nc (bumping from 2k
to 16k, though I'd'f liked to go larger)...

Also, you'd probably see even better performance by increasing the
size to 64k, though as with all things, this means more memory use
on smaller systems (though on smaller/slower systems, this may be
an even bigger win)...  This mostly just reduces number of context
switches...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: SSE in libthr

2015-03-28 Thread John-Mark Gurney
Eric van Gyzen wrote this message on Fri, Mar 27, 2015 at 17:43 -0400:
 On 03/27/2015 16:49, Rui Paulo wrote:
 
  Regarding your patch, I think we should disable even more, if possible.  
  How about:
 
  CFLAGS+=-mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3
 
 Yes, I was considering copying all of the similar flags that we use in the
 kernel.  That seems wise.  According to comments in sys/conf/kern.mk, only
 no-mmx and no-sse would be necessary, as they imply the others.
 
 dim@ raised the possibility of CPUTYPE=foo on i386, so I would also apply this
 change to i386.
 
 An updated patch is below.

We should probably add a $(CFLAGS_NOFPU) define and use that.. Then it
can be properly tweaked per compiler and per arch as necessary instead
of hardcoding the selection in each makefile...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: URGENT: RNG broken for last 4 months

2015-02-17 Thread John-Mark Gurney
Oliver Pinter wrote this message on Tue, Feb 17, 2015 at 23:27 +0100:
 On Tue, Feb 17, 2015 at 11:19 PM, Fabian Keil
 freebsd-lis...@fabiankeil.de wrote:
  John-Mark Gurney j...@funkthat.com wrote:
 
  If you are running a current kernel r273872 or later, please upgrade
  your kernel to r278907 or later immediately and regenerate keys.
 
  I tried ...
 
  start_init: trying /sbin/init
  118[20] Setting hostuuid: [...]
  118[20] Setting hostid: [...
  [20]
  [20]
  [20] Fatal trap 12: page fault while in kernel mode
  [20] cpuid = 1; apic id = 01
  [20] fault virtual address  = 0xf7ff1defb51c
  [20] fault code = supervisor read data, page not present
  [20] instruction pointer= 0x20:0x819eaed5
  [20] stack pointer  = 0x28:0xfe009397b890
  [20] frame pointer  = 0x28:0xfe009397b8d0
  [20] code segment   = base 0x0, limit 0xf, type 0x1b
  [20]= DPL 0, pres 1, long 1, def32 0, gran 1
  [20] processor eflags   = interrupt enabled, resume, IOPL = 0
  [20] current process= 43 (zfs)
  [...]
  ) at pcpu.h:219
  219 pcpu.h: No such file or directory.
  in pcpu.h
  (kgdb) where
  #0  doadump (textdump=Unhandled dwarf expression opcode 0x93
  ) at pcpu.h:219
  #1  0x8031539e in db_dump (dummy=value optimized out, 
  dummy2=Unhandled dwarf expression opcode 0x93
  ) at /usr/src/sys/ddb/db_command.c:533
  #2  0x80314e7c in db_command (cmd_table=0x0) at 
  /usr/src/sys/ddb/db_command.c:440
  #3  0x80314be4 in db_command_loop () at 
  /usr/src/sys/ddb/db_command.c:493
  #4  0x803177a0 in db_trap (type=value optimized out, 
  code=Unhandled dwarf expression opcode 0x93
  ) at /usr/src/sys/ddb/db_main.c:251
  #5  0x805ff8ee in kdb_trap (type=Unhandled dwarf expression opcode 
  0x93
  ) at /usr/src/sys/kern/subr_kdb.c:654
  #6  0x80889db9 in trap_fatal (frame=0xfe009397b7e0, eva=value 
  optimized out) at /usr/src/sys/amd64/amd64/trap.c:856
  #7  0x8088a131 in trap_pfault (frame=0xfe009397b7e0, 
  usermode=value optimized out) at /usr/src/sys/amd64/amd64/trap.c:678
  #8  0x8088976e in trap (frame=0xfe009397b7e0) at 
  /usr/src/sys/amd64/amd64/trap.c:426
  #9  0x8086cd82 in calltrap () at 
  /usr/src/sys/amd64/amd64/exception.S:235
  #10 0x819eaed5 in vdev_mirror_dva_select (zio=0xf80006549398, 
  p=-974039959) at 
  /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:317
  #11 0x819eae4a in vdev_mirror_preferred_child_randomize 
  (zio=0xf80006549398) at 
  /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:334
  #12 0x819eaba1 in vdev_mirror_child_select (zio=0xf80006549398) 
  at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:404
  #13 0x819ea177 in vdev_mirror_io_start (zio=0xf80006549398) at 
  /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:460
  #14 0x81a1d73d in zio_vdev_io_start (zio=0xf80006549398) at 
  /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2680
  #15 0x81a19c14 in zio_execute (zio=0xf80006549398) at 
  /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1499
  #16 0x81a18945 in zio_wait (zio=0xf80006549398) at 
  /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1523
  #17 0x81938db2 in arc_read (pio=0x0, spa=0xf8000634e000, 
  bp=0xf800065c5048, done=0x81937ae0 arc_getbuf_func, 
  private=0xf800065c9410, priority=ZIO_PRIORITY_SYNC_READ,
  zio_flags=128, arc_flags=0xfe009397c004, zb=0xfe009397bfe0) at 
  /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:3610
  #18 0x81964326 in dmu_objset_open_impl (spa=0xf8000634e000, 
  ds=0x0, bp=0xf800065c5048, osp=0xf800065c5008) at 
  /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:307
  #19 0x81991404 in dsl_pool_init (spa=0xf8000634e000, 
  txg=1056266109, dpp=0xf8000634e2e8) at 
  /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c:282
  #20 0x819c8b08 in spa_load_impl (spa=0xf8000634e000, 
  pool_guid=4830954193867998892, config=0xf80002599ee0, 
  state=SPA_LOAD_OPEN, type=SPA_IMPORT_EXISTING, mosconfig=0,
  ereport=0xfe009397c4e0) at 
  /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:2406
  #21 0x819c0987 in spa_load (spa=0xf8000634e000, 
  state=SPA_LOAD_OPEN, type=SPA_IMPORT_EXISTING, mosconfig=0) at 
  /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:2178
  #22 0x819bfda9 in spa_load_best (spa=0xf8000634e000, 
  state=SPA_LOAD_OPEN, mosconfig=0, max_request=18446744073709551615, 
  rewind_flags=1)
  at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:2903
  #23 0x819babe9 in spa_open_common (pool=0xfe0003232000 tank, 
  spapp

URGENT: RNG broken for last 4 months

2015-02-17 Thread John-Mark Gurney
If you are running a current kernel r273872 or later, please upgrade
your kernel to r278907 or later immediately and regenerate keys.

I discovered an issue where the new framework code was not calling
randomdev_init_reader, which means that read_random(9) was not returning
good random data.  read_random(9) is used by arc4random(9) which is
the primary method that arc4random(3) is seeded from.

This means most/all keys generated may be predictable and must be
regenerated.  This includes, but not limited to, ssh keys and keys
generated by openssl.  This is purely a kernel issue, and a simple
kernel upgrade w/ the patch is sufficient to fix the issue.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: URGENT: RNG broken for last 4 months

2015-02-17 Thread John-Mark Gurney
John-Mark Gurney wrote this message on Tue, Feb 17, 2015 at 09:37 -0800:
 If you are running a current kernel r273872 or later, please upgrade
 your kernel to r278907 or later immediately and regenerate keys.
 
 I discovered an issue where the new framework code was not calling
 randomdev_init_reader, which means that read_random(9) was not returning
 good random data.  read_random(9) is used by arc4random(9) which is
 the primary method that arc4random(3) is seeded from.
 
 This means most/all keys generated may be predictable and must be
 regenerated.  This includes, but not limited to, ssh keys and keys
 generated by openssl.  This is purely a kernel issue, and a simple
 kernel upgrade w/ the patch is sufficient to fix the issue.

It was brought to my attention (thanks Juli) that it might not be
clear that this issue does not effect any released version of FreeBSD.
It only effects people who run -current.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


PCIe hotplug, non-x86 hardware..

2015-02-14 Thread John-Mark Gurney
Hello,

I am working on adding PCIe Hot Plug to FreeBSD.  I have some basic
work already done, but I only have x86 hardware to test.  I was
wondering if anyone has any PCIe Hot Plug capabile hardware that isn't
x86, specificly, any sparc64 or powerpc hardware.

If so, what is your hardware, and would you be willing to run some
basic tests?

I've set reply-to so as not to spam the list.

Thanks.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: request for crypto hardware...

2015-02-06 Thread John-Mark Gurney
Benjamin Perrault wrote this message on Fri, Feb 06, 2015 at 18:41 -0800:
 I have a Soekris Net5501 ( glxsb ) w/ a vpn1411 card in it ( which is a hifn 
 based crypto board ) that you can have. And they are in SF, so, I believe, 
 fairly local for you. 
 
 Let me know if they are of interest.

Thanks, I've taken this offer off list.

  On Feb 6, 2015, at 6:35 PM, John-Mark Gurney j...@funkthat.com wrote:
  
  I have some plans to improve the opencrypto framework in FreeBSD later
  this year.  This will require invasive changes to the various drivers.
  So, I'd like to line up hardware/volunteers before then.
  
  If you would like to see your hardware tested and verified to work
  with the new changes, please contact me w/ either a donation of
  hardware, money to purchase hardware, or if you have hardware, that
  you volunteer to test changes.
  
  I currently have the following hardware:
  aesni
  
  I do not have the following hardware:
  hifn
  ubsec
  padlock (VIA C3, C7 and Eden)
  cesa (Marvell, missing man page)
  glxsb (AMD Geode LX, such as Sokris Net5501, missing man page)
  safe (SafeNet)
  sec (Freescale, missing man page)
  cryptocteon (Cavium Octeon, missing man page)
  nlmsec (mips/nlm/dev/sec/nlmsec.c, missing man page)
  rmisec (mips/rmi/dev/sec/rmisec.c, missing man page)

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


request for crypto hardware...

2015-02-06 Thread John-Mark Gurney
I have some plans to improve the opencrypto framework in FreeBSD later
this year.  This will require invasive changes to the various drivers.
So, I'd like to line up hardware/volunteers before then.

If you would like to see your hardware tested and verified to work
with the new changes, please contact me w/ either a donation of
hardware, money to purchase hardware, or if you have hardware, that
you volunteer to test changes.

I currently have the following hardware:
aesni

I do not have the following hardware:
hifn
ubsec
padlock (VIA C3, C7 and Eden)
cesa (Marvell, missing man page)
glxsb (AMD Geode LX, such as Sokris Net5501, missing man page)
safe (SafeNet)
sec (Freescale, missing man page)
cryptocteon (Cavium Octeon, missing man page)
nlmsec (mips/nlm/dev/sec/nlmsec.c, missing man page)
rmisec (mips/rmi/dev/sec/rmisec.c, missing man page)

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


  1   2   3   4   >