Re: The 11.1-RC3 can only boot and attach disks in "Safe mode", otherwise gets stuck attaching

[Mark Johnston]

On Tue, Jul 18, 2017 at 01:01:16AM +0200, Mark Martinec wrote:
> Upgrading 11.0-RELEASE-p11 to 11.1-RC3 using the usual freebsd-update 
> upgrade
> method I ended up with a system which gets stuck while trying to attach
> the second set of disks. This happened already after the first phase of
> the upgrade procedure (installing and re-booting with a new kernel).
> 
> The first set of disks (ada0 .. ada2) are attached successfully, also a
> cd0, but then when the first of the set of four (a regular spinning 
> disk)
> on an LSI controller is to be attached, the boot procedure just gets
> stuck there:
> 
>kernel: ada1: 300.000MB/s transfers (SATA 2.x, PIO4, PIO 8192bytes)
>kernel: ada1: Command Queueing enabled
>kernel: ada1: 305245MB (625142448 512 byte sectors)
>kernel: ada2 at ahcich6 bus 0 scbus8 target 0 lun 0
>kernel: ada2:  ATA8-ACS SATA 3.x device
>kernel: ada2: Serial Number OCZ-O1L6RF591R09Z5C8
>kernel: ada2: 300.000MB/s transfers (SATA 2.x, PIO4, PIO 8192bytes)
>kernel: ada2: Command Queueing enabled
>kernel: ada2: 114473MB (234441648 512 byte sectors)
>kernel: ada2: quirks=0x1<4K>
>kernel: da0 at mps0 bus 0 scbus0 target 2 lun 0
> 
> (stuck here, keyboard not responding, fans rising their pitch,
>   presumably CPU is spinning)

Are you able to break into the debugger at this point? Try setting
debug.kdb.break_to_debugger=1 and debug.kdb.alt_break_to_debugger=1 at
the loader prompt, and hit the break key, or the key sequence
 ~ ctrl-b once the hang occurs. At the debugger prompt, try
"bt" and "show allpcpu" to start.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: stack_guard hardening bsdinstall option in STABLE and 11.1

[Vlad K.]

On 2017-07-18 00:09, Mark Millard wrote:

(Although I expect Konstantin Belousov's note here is
the first public description of the problem's details.)


Thanks for explaining the problem. I guess this was the reason why I 
failed to parse kib's reply, this was the first bit of info I 
encountered on that patch being effectively "broken" that way.




I agree that you did not get an answer for the other
part:


I simply asked if it's safe to assume the sysctl to be an integer in



11.1



I've not gone through any draft 11.1-release code to
check.


It appears to be, the code is MFC'd with (if I'm correct) r320666. I've 
ran some tests in -RC3 and indeed it works, though probably for the 
reason you explained above (guard page eating into the stack), raising 
the stack_guard_pages sufficiently high (eg. 512 pages like the 
bsdinstaller in CURRENT defaults to) crashes threaded programs.


If that is so, though, I wonder why it's not reverted, or at least the 
sysctl temporarily patched to remain boolean (or turned off completely). 
And the bsdinstaller option in CURRENT now essentially enables buggy and 
unstable behavior. If this is a known issue, why default to it in 
CURRENT.



Anyway thanks for taking time to explain, this answers my questions.



--
Vlad K.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

The 11.1-RC3 can only boot and attach disks in "Safe mode", otherwise gets stuck attaching

[Mark Martinec]

Upgrading 11.0-RELEASE-p11 to 11.1-RC3 using the usual freebsd-update 
upgrade

method I ended up with a system which gets stuck while trying to attach
the second set of disks. This happened already after the first phase of
the upgrade procedure (installing and re-booting with a new kernel).

The first set of disks (ada0 .. ada2) are attached successfully, also a
cd0, but then when the first of the set of four (a regular spinning 
disk)

on an LSI controller is to be attached, the boot procedure just gets
stuck there:

  kernel: ada1: 300.000MB/s transfers (SATA 2.x, PIO4, PIO 8192bytes)
  kernel: ada1: Command Queueing enabled
  kernel: ada1: 305245MB (625142448 512 byte sectors)
  kernel: ada2 at ahcich6 bus 0 scbus8 target 0 lun 0
  kernel: ada2:  ATA8-ACS SATA 3.x device
  kernel: ada2: Serial Number OCZ-O1L6RF591R09Z5C8
  kernel: ada2: 300.000MB/s transfers (SATA 2.x, PIO4, PIO 8192bytes)
  kernel: ada2: Command Queueing enabled
  kernel: ada2: 114473MB (234441648 512 byte sectors)
  kernel: ada2: quirks=0x1<4K>
  kernel: da0 at mps0 bus 0 scbus0 target 2 lun 0

(stuck here, keyboard not responding, fans rising their pitch,
 presumably CPU is spinning)

(instead of the normal continuation like:
  kernel: da0:  Fixed Direct Access SPC-4 SCSI 
device

  kernel: da0: Serial Number 
  kernel: da0: 600.000MB/s transfers
  kernel: da0: Command Queueing enabled
  kernel: da0: 1907729MB (3907029168 512 byte sectors)
)

The controller for da0 .. da3 is an LSI:

  kernel: mps0:  port 0x4000-0x40ff 
mem 0xd174-0xd1743fff,0xd130-0xd133 irq 16 at device 0.0 on 
pci1

  kernel: mps0: Firmware: 14.00.01.00, Driver: 21.02.00.00-fbsd
  kernel: mps0: IOCCapabilities: 
185c

[...]
  kernel: mps0: SAS Address for SATA device = a4a4843003d0cf79
  kernel: mps0: SAS Address from SATA device = a4a4843003d0cf79
  kernel: mps0: SAS Address for SATA device = d3d48904eddff0d5
  kernel: mps0: SAS Address from SATA device = d3d48904eddff0d5
[...]
  kernel: mps0: SAS Address for SATA device = 2a021c07585c665b
  kernel: mps0: SAS Address from SATA device = 2a021c07585c665b
  kernel: mps0: SAS Address for SATA device = 2a021c0758637b7c
  kernel: mps0: SAS Address from SATA device = 2a021c0758637b7c

This host in this configuration worked perfectly well with 11.0 and
many older versions of the OS.

After some frustration I found out that the system can boot fine
if a boot loader option "Safe mode" is set. This way I successfully
finished the upgrade procedure (installing world).

Playing with loader options that the "Safe mode" turns on
( /boot/menu-commands.4th ) it seems that kern.smp.disabled=1
is the crucial option, although my attempts at ruling out remaining
options of the "Safe mode" turned out inconclusive - perhaps there
is some random/race involved. Anyway, in "Safe mode" the machine
always boots normally and attaches all disks.

This experience is much like described in:
  https://forums.freebsd.org/threads/56524/
where the poster ended up disabling SMP to be able to have a working 
host.


It is also somewhat similar to:
  
https://lists.freebsd.org/pipermail/freebsd-hackers/2017-July/051258.html

where a FreeBSD 11.1 prerelease only boots on a single-CPU AWS host,
but fails to boot on a 2-core CPU, with various symptoms, including:
  ( 
https://lists.freebsd.org/pipermail/freebsd-hackers/2017-July/051260.html 
)

  Feeding entropy: .
  spin lock 0x80db45c0 (smp rendezvous) held by 
0xf80004378560

  (tid 100074) too long
  timeout stopping cpus
  panic: spin lock held too long

Please advise, thanks
  Mark
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: stack_guard hardening bsdinstall option in STABLE and 11.1

[Mark Millard]

Vlad K. vlad-fbsd at acheronmedia.com wrote on
Mon Jul 17 15:03:11 UTC 2017 :

> I also asked why wasn't the bsdinstall-er option change 
> MFC'd after 1 day, two weeks ago, whether it's by omission, simply 
> ENOTIME, or something else...

Given what Konstantin Belousov described (default
stack space sizes and apparently guard pages eat
into stack space instead of the overall space being
bigger by the guard size), I think that would explain
not moving from CURRENT: it was known to be a problem.
(Although I expect Konstantin Belousov's note here is
the first public description of the problem's details.)

I agree that you did not get an answer for the other
part:

> I simply asked if it's safe to assume the sysctl to be an integer in 

> 11.1


I've not gone through any draft 11.1-release code to
check.

===
Mark Millard
markmi at dsl-only.net

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: stack_guard hardening bsdinstall option in STABLE and 11.1

[Vlad K.]

On 2017-07-17 16:11, Glen Barber wrote:


kib gave feedback on this in an earlier reply (which I missed before
replying myself).



Neither of which answered my questions, I'm sorry. My question was not 
about stack sizes in 32 or 64 bit installations, nor about the quality 
of the fix (if I parse the rm libtrh comment correctly).


I simply asked if it's safe to assume the sysctl to be an integer in 
11.1 (I'm guessing yes looking at the commits to STABLE, but wanted to 
be sure), and I also asked why wasn't the bsdinstall-er option change 
MFC'd after 1 day, two weeks ago, whether it's by omission, simply 
ENOTIME, or something else...



--
Vlad K.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: stack_guard hardening bsdinstall option in STABLE and 11.1

[Glen Barber]

On Mon, Jul 17, 2017 at 03:47:08PM +0200, Vlad K. wrote:
> On 2017-07-17 15:33, Glen Barber wrote:
> > 
> > No, this is not available in the 11.1 installer.
> > 
> 
> Thanks but that's why I asked why's that. r320674 said MFC after 1 day. Is
> it too late for 11.1-RELEASE, so it'll be applied to 11-STABLE, or is there
> another reason?
> 
> If its' too late, does that mean it's too late for the installer, but the
> new stack_guard code is there in STABLE and I am guessing will be part of
> 11.1, so we can assume the sysctl to be an integer (as opposed to
> enable/disable semantics of the sysctl in 11.0)? In other words, is it safe
> to ramp up the gap size in 11.1?
> 

kib gave feedback on this in an earlier reply (which I missed before
replying myself).

Glen



signature.asc
Description: PGP signature

Oracle SCM users list.

[fiona . jackson]

 

 

Hi,

 

We have updated our Oracle
SCM users list and thought you would be interested in receiving fresh and
accurate contacts for your marketing campaign. We provide data across the globe
so you can target your audience and reach to a wider market and increase your
lead flow.

 

We can filter the list according to your requirement as we
have a dedicated data team of 200
members who verify the contacts and keep them updated.

 

We can also provide you with other technologies such as SAP
Logistics, Infor SCM, Epicor WMS and many more. let me know your requirement
and I will get back to you with more information regarding the same.

 

Thank you for your valuable time look forward to hear from
you.

 

Regards

Fiona Jackson  

To
OPT-OUT please reply Leave OUT in subject line.

 

 

 

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: stack_guard hardening bsdinstall option in STABLE and 11.1

[Vlad K.]

On 2017-07-17 15:33, Glen Barber wrote:


No, this is not available in the 11.1 installer.

Glen


Thanks but that's why I asked why's that. r320674 said MFC after 1 day. 
Is it too late for 11.1-RELEASE, so it'll be applied to 11-STABLE, or is 
there another reason?


If its' too late, does that mean it's too late for the installer, but 
the new stack_guard code is there in STABLE and I am guessing will be 
part of 11.1, so we can assume the sysctl to be an integer (as opposed 
to enable/disable semantics of the sysctl in 11.0)? In other words, is 
it safe to ramp up the gap size in 11.1?



--
Vlad K.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: stack_guard hardening bsdinstall option in STABLE and 11.1

[Glen Barber]

On Mon, Jul 17, 2017 at 11:54:06AM +0200, Vlad K. wrote:
> Hello list,
> 
> the stack_guard hardening option in bsdinstall is now setting 512 pages of
> it in CURRENT, as of r320674. It's said to MFC after 1 day (on Jul 5th), but
> STABLE hasn't got it yet. Is this simply an omission (understandable as the
> RELEASE is being prepared so things are a bit hectic I guess), or is there
> another reason?
> 
> Can we assume that in 11.1 the sysctl is integer and can we safely set >1
> number of pages, say 512 like the installer in CURRENT suggests?
> 

No, this is not available in the 11.1 installer.

Glen



signature.asc
Description: PGP signature

Re: stack_guard hardening bsdinstall option in STABLE and 11.1

[Konstantin Belousov]

On Mon, Jul 17, 2017 at 11:54:06AM +0200, Vlad K. wrote:
> Hello list,
> 
> the stack_guard hardening option in bsdinstall is now setting 512 pages 
> of it in CURRENT, as of r320674. It's said to MFC after 1 day (on Jul 
> 5th), but STABLE hasn't got it yet. Is this simply an omission 
> (understandable as the RELEASE is being prepared so things are a bit 
> hectic I guess), or is there another reason?
> 
> Can we assume that in 11.1 the sysctl is integer and can we safely set 
>  >1 number of pages, say 512 like the installer in CURRENT suggests?

Default stack size on 32bit platforms is 2M.  I left it to you as an
excercise to guess what happens with the setting applied.

For 64bit machines, default stack size is 4M, so there the failure mode is
somewhat more involved.

Anyway, this option is almost equivalent to executing 'rm /lib/libthr.so.3',
perhaphs rm is even beter.  SECURITY !  HARDENING !
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

stack_guard hardening bsdinstall option in STABLE and 11.1

[Vlad K.]

Hello list,

the stack_guard hardening option in bsdinstall is now setting 512 pages 
of it in CURRENT, as of r320674. It's said to MFC after 1 day (on Jul 
5th), but STABLE hasn't got it yet. Is this simply an omission 
(understandable as the RELEASE is being prepared so things are a bit 
hectic I guess), or is there another reason?


Can we assume that in 11.1 the sysctl is integer and can we safely set 
>1 number of pages, say 512 like the installer in CURRENT suggests?


Thanks!



--
Vlad K.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"