Re: Intel Xeon D-2146NT SoC Support

[O'Connor, Daniel]

> On 7 Feb 2019, at 10:58, Lyndon Nerenberg  wrote:
> 
> [ I asked this on -hardware but didn't get any responses, so I'm
>  widening the scope a bit. ]
> 
> We're looking a buying a Supermicro system with the X11SDV-8C-TP8F
> motherboard.  This uses a Xeon D-2146NT SoC.  It's not clear what
> the embedded chipset is, so it's proving a bit tricky to confirm
> whether or not the board will get along with 11.2.

If you go to OS compatibility and then go Skylake-D SoC you get to 
https://www.supermicro.com/support/resources/OS/skylake-d.cfm
(This is not really obvious though! I looked up the CPU in ark.intel.com to 
find its codename first..)

It says FreeBSD 11 works but no 11.1, 11.2 or 12.0. Personally I suspect they 
*do* work fine just they didn't bother to test them.

I certainly would be very surprised if 11.2 *didn't* work if 11.0 did (and it 
would be a regression and should be fixed) however it's not my money so.. :)

--
Daniel O'Connor
"The nice thing about standards is that there
are so many of them to choose from."
 -- Andrew Tanenbaum


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Intel Xeon D-2146NT SoC Support

[Lyndon Nerenberg]

[ I asked this on -hardware but didn't get any responses, so I'm
  widening the scope a bit. ]

We're looking a buying a Supermicro system with the X11SDV-8C-TP8F
motherboard.  This uses a Xeon D-2146NT SoC.  It's not clear what
the embedded chipset is, so it's proving a bit tricky to confirm
whether or not the board will get along with 11.2.

In particular, we need to ensure that we can talk to each of the
12 SATA ports as a standalone (JBOD) disk (this will be a ZFS
fileserver), and that the SFP+ ports will run with 10 Gb/s optics.

If anyone is running this board (or anything else running that D-2146NT
chip) I'd appreciate hearing from you.

FWIW, the system we're looking at is the SSG-5019D8-TR12P:

  https://www.supermicro.com/products/system/1U/5019/SSG-5019D8-TR12P.cfm

Thanks!

--lyndon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: 9211 (LSI/SAS) issues on 11.2-STABLE

[Karl Denninger]

On 2/6/2019 09:18, Borja Marcos wrote:
>> On 5 Feb 2019, at 23:49, Karl Denninger  wrote:
>>
>> BTW under 12.0-STABLE (built this afternoon after the advisories came
>> out, with the patches) it's MUCH worse.  I get the same device resets
>> BUT it's followed by an immediate panic which I cannot dump as it
>> generates a page-fault (supervisor read data, page not present) in the
>> mps *driver* at mpssas_send_abort+0x21.
>> This precludes a dump of course since attempting to do so gives you a
>> double-panic (I was wondering why I didn't get a crash dump!); I'll
>> re-jigger the box to stick a dump device on an internal SATA device so I
>> can successfully get the dump when it happens and see if I can obtain a
>> proper crash dump on this.
>>
>> I think it's fair to assume that 12.0-STABLE should not panic on a disk
>> problem (unless of course the problem is trying to page something back
>> in -- it's not, the drive that aborts and resets is on a data pack doing
>> a scrub)
> It shouldn’t panic I imagine.
>
> mps0: Sending reset from mpssas_send_abort for target ID 37
>
>>> 0x06  =  =   =  ===  == Transport Statistics (rev 1) ==
>>> 0x06  0x008  4   6  ---  Number of Hardware Resets
>>> 0x06  0x010  4   0  ---  Number of ASR Events
>>> 0x06  0x018  4   0  ---  Number of Interface CRC Errors
>>> |||_ C monitored condition met
>>> ||__ D supports DSN
>>> |___ N normalized value
>>>
>>> 0x06  0x008  4   7  ---  Number of Hardware Resets
>>> 0x06  0x010  4   0  ---  Number of ASR Events
>>> 0x06  0x018  4   0  ---  Number of Interface CRC Errors
>>> |||_ C monitored condition met
>>> ||__ D supports DSN
>>> |___ N normalized value
>>>
>>> Number of Hardware Resets has incremented.  There are no other errors shown:
> What is _exactly_ that value? Is it related to the number of resets sent from 
> the HBA
> _or_ the device resetting by itself?
Good question.  What counts as a "reset"; UNIT ATTENTION is what the
controller receives but whether that's a power reset, a reset *command*
from the HBA or a firmware crash (yikes!) in the disk I'm not certain.
>>> I'd throw possible shade at the backplane or cable /but I have already
>>> swapped both out for spares without any change in behavior./
> What about the power supply? 
>
There are multiple other devices and the system board on that supply
(and thus voltage rails) but it too has been swapped out without
change.  In fact at this point other than the system board and RAM
(which is ECC, and is showing no errors in the system's BMC log)
/everything /in the server case (HBA, SATA expander, backplane, power
supply and cables) has been swapped for spares.  No change in behavior.

Note that with 20.0.7.0 firmware in the HBA instead of a unit attention
I get a *controller* reset (!!!) which detaches some random number of
devices from ZFS's point of view before it comes back up (depending on
what's active at the time) which is potentially catastrophic if it hits
the system pool.  I immediately went back to 19.0.0.0 firmware on the
HBA; I had upgraded to 20.0.7.0 since there had been good reports of
stability with it when I first saw this, thinking there was a drive
change that might have resulted in issues with it when running 19.0
firmware on the card.

This system was completely stable for over a year on 11.1-STABLE and in
fact hadn't been rebooted or logged a single "event" in over six months;
the problems started immediately upon upgrade to 11.2-STABLE and
persists on 12.0-STABLE.  The disks in question haven't changed either
(so it can't be a difference in firmware that is in a newer purchased
disk, for example.)

I'm thinking perhaps *something* in the codebase change made the HBA ->
SAS Expander combination trouble where it wasn't before; I've got a
couple of 16i HBAs on the way which will allow me to remove the SAS
expander to see if that causes the problem to disappear.  I've got a
bunch of these Lenovo expanders and have been using them without any
sort of trouble in multiple machines; it's only when I went beyond 11.1
that I started having trouble with them.

-- 
Karl Denninger
k...@denninger.net 
/The Market Ticker/
/[S/MIME encrypted email preferred]/


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Request for more intelligent local port allocation algorithm

[David King via freebsd-stable]

Just to add to this, if anyone is doing some work on the outbound tcp
connection, could they also have a look at the bug here
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=210726

Thanks!

On Wed, 6 Feb 2019 at 15:15, Paul  wrote:

> Hi dev team,
>
> It's not a secret that when application is trying to establish new TCP
> connection, without
> first binding a socket to specific local interface address, OS handles
> that automatically.
> Unfortunately there is a catch, that lies in a different logic of local
> port allocation:
> (1) when socket is bound before connect() vs (2) when it is not. When
> allocating the port
> in in_pcb_lport() by checking whether different ports are free, using
> in_pcblookup_local(),
> the behaviour is following:
>
> (1) Bound, ie laddr is assigned with specific address:
> Port is considered occupied only if there is a PCBs that matches both
> laddr and lport
>
> (2) Not bound, ie laddr == INADDR_ANY:
> Port is considered occupied if there is any PCBs that only matches
> lport. What this
> means is that in order to allocate a port none of the all available
> local addresses
> should have it allocated, even though this requirement is ridiculous,
> since we are
> allocating only one PCB
>
> Looking though the code, it seems that (2) is due to the fact that
> tcp_connect() first
> allocates the port, indirectly through the call to in_pcbbind() and only
> then allocates
> the actual local address, also indirectly, though the call to
> in_pcbconnect_setup(), that
> in turn calls in_pcbladdr(). So, probably, in order to guarantee that
> in_pcbconnect_setup()
> will not fail we make sure that all range of local addresses are
> available, no matter
> which one of them is actually selected by in_pcbladdr()?
>
> In real world, this creates serious problems for servers that have a lot
> of outgoing
> connections, for example nginx proxy with a lot of open HTTP2 connections.
> In order to
> avoid this limitation we have created workarounds within the nginx config
> as well as
> within our  own software, basically by having 50 local addresses and only
> following the
> scenario (1). Alas, all of the built-in Unix utilities as well as other
> software always
> follow scenario (2). As the result given large number of connections there
> may be points
> in time, when whole range of ports is occupied by at least one local
> address. Even worse is
> the outcome of such condition: when in_pcb_lport() travels over the range
> of possible port
> numbers, making myriad of calls to in_pcblookup_local(), some  kind of
> important lock is
> being held withing the kernel. So important that it leads to a complete
> lock of the system.
> Even the direct terminal access is not available: it is not responsive.
> The more calls to
> connect through scenario (2) there are the longer it takes the system to
> unfreeze. Given
> some circumstances, the only option is hard reset.
>
> Is it possible to somehow update the code that does connect via scenario
> (2) to enable
> more intelligent port allocation, like for example allocating local
> address and port simultaneously
>
> ___
> freebsd-stable@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
>
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: 9211 (LSI/SAS) issues on 11.2-STABLE

[Borja Marcos]

> On 5 Feb 2019, at 23:49, Karl Denninger  wrote:
> 
> BTW under 12.0-STABLE (built this afternoon after the advisories came
> out, with the patches) it's MUCH worse.  I get the same device resets
> BUT it's followed by an immediate panic which I cannot dump as it
> generates a page-fault (supervisor read data, page not present) in the
> mps *driver* at mpssas_send_abort+0x21.

> This precludes a dump of course since attempting to do so gives you a
> double-panic (I was wondering why I didn't get a crash dump!); I'll
> re-jigger the box to stick a dump device on an internal SATA device so I
> can successfully get the dump when it happens and see if I can obtain a
> proper crash dump on this.
> 
> I think it's fair to assume that 12.0-STABLE should not panic on a disk
> problem (unless of course the problem is trying to page something back
> in -- it's not, the drive that aborts and resets is on a data pack doing
> a scrub)

It shouldn’t panic I imagine.

 mps0: Sending reset from mpssas_send_abort for target ID 37


>> 0x06  =  =   =  ===  == Transport Statistics (rev 1) ==
>> 0x06  0x008  4   6  ---  Number of Hardware Resets
>> 0x06  0x010  4   0  ---  Number of ASR Events
>> 0x06  0x018  4   0  ---  Number of Interface CRC Errors
>> |||_ C monitored condition met
>> ||__ D supports DSN
>> |___ N normalized value
>> 
>> 0x06  0x008  4   7  ---  Number of Hardware Resets
>> 0x06  0x010  4   0  ---  Number of ASR Events
>> 0x06  0x018  4   0  ---  Number of Interface CRC Errors
>> |||_ C monitored condition met
>> ||__ D supports DSN
>> |___ N normalized value
>> 
>> Number of Hardware Resets has incremented.  There are no other errors shown:

What is _exactly_ that value? Is it related to the number of resets sent from 
the HBA
_or_ the device resetting by itself?

>> I'd throw possible shade at the backplane or cable /but I have already
>> swapped both out for spares without any change in behavior./

What about the power supply? 





Borja.


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Request for more intelligent local port allocation algorithm

[Paul]

Hi dev team,

It's not a secret that when application is trying to establish new TCP 
connection, without
first binding a socket to specific local interface address, OS handles that 
automatically.
Unfortunately there is a catch, that lies in a different logic of local port 
allocation: 
(1) when socket is bound before connect() vs (2) when it is not. When 
allocating the port  
in in_pcb_lport() by checking whether different ports are free, using 
in_pcblookup_local(),
the behaviour is following:

(1) Bound, ie laddr is assigned with specific address: 
Port is considered occupied only if there is a PCBs that matches both laddr 
and lport

(2) Not bound, ie laddr == INADDR_ANY: 
Port is considered occupied if there is any PCBs that only matches lport. 
What this  
means is that in order to allocate a port none of the all available local 
addresses  
should have it allocated, even though this requirement is ridiculous, since 
we are 
allocating only one PCB

Looking though the code, it seems that (2) is due to the fact that 
tcp_connect() first 
allocates the port, indirectly through the call to in_pcbbind() and only then 
allocates
the actual local address, also indirectly, though the call to 
in_pcbconnect_setup(), that
in turn calls in_pcbladdr(). So, probably, in order to guarantee that 
in_pcbconnect_setup()
will not fail we make sure that all range of local addresses are available, no 
matter 
which one of them is actually selected by in_pcbladdr()?

In real world, this creates serious problems for servers that have a lot of 
outgoing 
connections, for example nginx proxy with a lot of open HTTP2 connections. In 
order to 
avoid this limitation we have created workarounds within the nginx config as 
well as 
within our  own software, basically by having 50 local addresses and only 
following the 
scenario (1). Alas, all of the built-in Unix utilities as well as other 
software always  
follow scenario (2). As the result given large number of connections there may 
be points
in time, when whole range of ports is occupied by at least one local address. 
Even worse is  
the outcome of such condition: when in_pcb_lport() travels over the range of 
possible port 
numbers, making myriad of calls to in_pcblookup_local(), some  kind of 
important lock is 
being held withing the kernel. So important that it leads to a complete lock of 
the system.
Even the direct terminal access is not available: it is not responsive. The 
more calls to 
connect through scenario (2) there are the longer it takes the system to 
unfreeze. Given 
some circumstances, the only option is hard reset.

Is it possible to somehow update the code that does connect via scenario (2) to 
enable
more intelligent port allocation, like for example allocating local address and 
port simultaneously  

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: More CARP issues under 12

[Pete French]

On 06/02/2019 12:16, Andrey V. Elsukov wrote:

Hi,

this doesn't look very useful.
Do you have some specificity with this host except carp? Some
modifications to kernel config, lagg, jails, etc.


No, none of those. Its a supermicro motherboard, runs FreeBSD
GENERIC and mysql+redis on top, thats it. The only oddity is
carp (used to fail over the redis). but the panic happens when
I disable carp and have removed all the ports too. My only customisation 
to the build is to disable sendmail and lpr.


We do use geli for the dirves, and load aesni as a module as well to 
speed that up.


loader.conf below:

kern.geom.label.disk_ident.enable=0
kern.geom.label.gptid.enable=0

ahci_load="YES"
console="comconsole"

aesni_load="YES"
cryptodev_load="YES"
geom_eli_load="YES"
carp_load="YES"

zfs_load="YES"
vfs.zfs.arc_max="1G"
vfs.zfs.prefetch_disable="1"
vfs.zfs.txg.timeout="5"
vfs.zfs.vdev.cache.size="10M"
vfs.zfs.vdev.cache.max="10M"

rc.conf below

geli_enable="YES"
geli_autodetach="NO"
geli_devices="ada0p4 ada1p4"

hostname="serpentine-passive.telehouse-internal.ingresso.co.uk"

ifconfig_igb0="inet 10.32.10.4/16"
ifconfig_igb0_ipv6="inet6 2a02:1658:1:2:e550::4/64"
ifconfig_igb0_alias0="inet 10.32.10.8/16 vhid 80 advskew 160 pass 
redacted"

defaultrouter="10.32.10.6"
ipv6_defaultrouter="2a02:1658:1:2:e550::6"

ifconfig_igb1="down"

pf_enable="NO"
pf_rules="/usr/local/etc/pf.conf"

redis_enable="YES"
stunnel_enable="YES"

mysql_enable="YES"
mysql_dbdir="/usr/home/mysql/data"

tsw_redis_capture_enable="YES"
tsw_redis_capture_if="igb0"

datadog_enable="YES"
datadog_user="root"
datadog_chdir="/usr/local/datadog"

sshd_enable="YES"
named_enable="YES"
zfs_enable="YES"
ntpd_enable="YES"

syslogd_enable="NO"
syslog_ng_enable="YES"

exim_enable="YES"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

nfs_server_enable="NO"
nfs_client_enable="YES"
nfsv4_server_enable="NO"
nfsuserd_enable="YES"
rpcbind_enable="YES"
rpc_lockd_enable="YES"
rpc_lockd_flags="-p 819"
rpc_statd_enable="YES"
rpc_statd_flags="-p 823"
mountd_enable="NO"

fluentd_enable="YES"

The tsw_redis_capture script just set the carp to MASTER if redis is 
enabled - means if the machine boots without redis running then carp 
wont grap the address anyway.


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: More CARP issues under 12

[Andrey V. Elsukov]

On 05.02.2019 18:06, Pete French wrote:
> The branch and revision is 12.0-STABLE r343538 GENERIC
> 
>> # kgdb
>>
>> (kgdb) list *ether_output+0x6b6
> 
> trying to do this on the actual box is hard, as it panics, but on another
> machine running the same build I get this, which should suffice if you
> are just interested in seeing the line in the source code ?
> 
> (kgdb)  list *ether_output+0x6b6
> 0x80ca1526 is in ether_output (/usr/src/sys/net/if_ethersubr.c:435).
> 430 if (m == NULL)
> 431 return (0);
> 432 }
> 433
> 434 /* Continue with link-layer output */
> 435 return ether_output_frame(ifp, m);
> 436 }
> 437
> 438 static bool
> 439 ether_set_pcp(struct mbuf **mp, struct ifnet *ifp, uint8_t pcp)

Hi,

this doesn't look very useful.
Do you have some specificity with this host except carp? Some
modifications to kernel config, lagg, jails, etc.

-- 
WBR, Andrey V. Elsukov



signature.asc
Description: OpenPGP digital signature