Re: [smartos-discuss] Re: Weird disk performance issues with newer images

[Michal Nowak]

On 07/25/18 08:49 PM, Len Weincier wrote:

Hi

I see this commit and the hosts where we see issues have 2 NVMe devs 
used as slogs if that helps


https://github.com/joyent/illumos-joyent/commit/f78cdc34af236a6199dd9e21376f4a46348c0d56

Len


On 25 Jul 2018, at 16:58, Len Weincier > wrote:


Hi

We a very strange situation trying to upgrade to a newer smartos image 
where the disk I/O is *very* slow.


I have been working through the released images and the last one that 
works 100% is 20180329T002644Z


From 20180412T003259Z onwards, the release with the new zfs features 
like spacemaps etc, the hosts become unusable in terms of disk i/o


In our testing with the lab machine with only 128G ram we see no 
pathologies.


Hosts are running ALL SSDs (RAIDZ2), and Intel Gold 6150 x2 processors 
on an SMC X11DPH-T board..
The lab machine with 128GB RAM has exactly the same processors, board, 
and SSD-only setup - except for RAM..


On a production machine with 768G ram and the newer image for eg zfs 
create -V 10G zones/test takes 2 minutes while at the same time iostat 
is showing the disks as relatively idle (%b = 10)


For example inside an ubuntu kvm with postgres we are seeing 40% wait 
time for any disk i/o and there are only 2 vm's running, underlying 
disks essentially idle.


Is there anything we can look at to get to the bottom of this as it 
pretty critical and affecting our customers


Any help appreciated
Thanks
Len


The log 
(https://us-east.manta.joyent.com/Joyent_Dev/public/SmartOS/smartos.html#20180412T003259Z) 
also says "Implement KPTI". Does the problem go away when you boot with 
kpti=0? I've seen disk IO going haywire for very old CPU without 
process-context identifiers (PCID).


Michal


---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: https://www.listbox.com

Re: [smartos-discuss] anyone know a SmartOS friendly Mini-ITX motherboard?

[Michal Nowak]

On 07/21/18 01:51 PM, Robert Fisher wrote:
I've given up on the idea of any Illumos flavour ever running on my 
little home server, so I'm planning to swap out the main board.


The backstory of this is that I successfully ran Solaris 11 on it 
(encrypted ZFS -- I'm paranoid) and when I upgraded the BIOS (I know, 
never upgrade the BIOS, but I was hoping to fix a NIC oddity) it no 
longer booted. I found if I disabled all but one core, Solaris would 
still boot, and I still run it that way, which is pretty unsatisfactory. 
SmartOS will not boot whatever I do. All other unix-like OSes run fine 
across all cores. I have posted about this on the list before, but to no 
avail.


I don't want to buy a whole new machine, so I need a Mini ITX 
motherboard that'll take an i5 6400 (so 6th gen) CPU. Anyone got 
anything that's known to work? The one that doesn't is an ASUS H110I-Plus.



thanks in advance


Rob


Hi Rob,

I don't know of a suitable mini ITX motherboard but it seems that there 
were three BIOS updates since your original February post regarding the 
motherboard. Perhaps you can take a chance updating to the latest BIOS? 
Or perhaps you can downgrade to a known working one?


Good luck,
Michal


---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: https://www.listbox.com

Re: [smartos-discuss] HYPER-V

[Michal Nowak]

On 04/11/2018 05:43 PM, Jeremy wrote:Hello,
  I am trying to find more info on hyper-v support.  I see I
  can use a legacy nic to get networking up and running, but
  it seems terribly slow. 
Is there any work for this?
  Thank you.
-Jeremy
Hello Jeremy.
As far as I know there's no Hyper-V-specific support in SmartOS, no
"enlightened" drivers. I remember I ran OpenIndiana on Windows 10
Pro Hyper-V and the only devices I was able to use were those you
yourself found (emulated IDE hard disk and CDROM, emulated 100 Mbit
NIC, ...). Delphix is said is working on proper support for their
products (they target MS Azure), but I am not aware if they released
something already (or if they intend to open the code at some
point).
Probably the best source of information is this email thread:
https://illumos.topicbox.com/groups/developer/T8c8e2aa6b9b46c1f and
these slides:
https://www.bsdcan.org/2017/schedule/events/856.en.html
Michal
  smartos-discuss | Archives
 | Modify
 Your Subscription

Re: [smartos-discuss] KPTI testing help wanted (PCID edition)

[Michal Nowak]

On Wednesday, March 7, 2018 at 2:07 PM, John Levon <le...@movementarian.org> 
wrote:
> On Wed, Mar 07, 2018 at 01:58:16PM +0100, Michal Nowak wrote:
> 
>> Just filed: https://github.com/joyent/smartos-live/issues/768
>>
>> I think I should test if it also fails on EHCI and with latest non-KPTI
>> SmartOS image.
> 
> That would be very useful.
> 
> Also, can you try adding to the end of the grub boot line (where it says
> smartos=true):
> 
> kpti=0
> 
> This should turn off KPTI itself (although there are still changes).
> 
> regards
> john

Thanks for the pointer to 'kpti=0'.

The panic is only present when xHCI is enabled and the USB Ethernet adapter is 
present. If I switch to EHCI or the adapter is not present, the panic does not 
happen. KPTI on/off is not changing anything.

Will mention this findings in the ticket as well.

Thanks!
Michal


---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] KPTI testing help wanted (PCID edition)

[Michal Nowak]

On Tuesday, March 6, 2018 at 10:29 AM, John Levon <le...@movementarian.org> 
wrote:
> On Tue, Mar 06, 2018 at 07:21:45am +0100, Michal Nowak wrote:
> 
>> >>https://us-east.manta.joyent.com/arekinath/public/kpti/20180221T003551Z/platform-20180221T003551Z.tgz
>> >>https://us-east.manta.joyent.com/arekinath/public/kpti/20180221T003551Z/platform-20180221T003551Z.iso
>> >>https://us-east.manta.joyent.com/arekinath/public/kpti/20180221T003551Z/platform-20180221T003551Z.usb.bz2
>> >
>> >with the USB image I am unable to use virtual console both on my
>> >Toshiba Satellite L855 laptop (Ivy Bridge) and as well as on Linux
>> >KVM (CPU is set as host-passthrough). The former KPTI image's
>> >virtual console works in both cases.
>>
>> Now I see what is my problem. When I boot this and the newest image
>> (platform-20180305T101513Z), serial console in form of `os_console`
>> variable is set to `ttyb`, but I don't have any serial console on
>> the laptop. When I manually set it to `vga`, I am all good. (On
>> Linux KVM it's probably the same thing.)
> 
> Apologies, yes, the builds default to an os_console setting of ttyb; I
> should have mentioned that.
> 
> regards
> john

Hi.

Just filed: https://github.com/joyent/smartos-live/issues/768

I think I should test if it also fails on EHCI and with latest non-KPTI SmartOS 
image.

Let me know, if you need more information.

Thanks,
Michal


---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] KPTI testing help wanted (PCID edition)

[Michal Nowak]

On 03/01/18 07:13 AM, Michal Nowak wrote:

On 02/23/2018 03:17 AM, Alex Wilson wrote:

Hi all,

Same deal as before, I would like to solicit testers for a DEBUG image
of KPTI with PCID support this time. I am particularly interested in
anyone who has a Nehalem or Westmere era Xeon to try this one out, as
well as anyone who can put significant load on it. Quite a few of the
bugs we've found so far have only been visible when all the CPUs were
kept busy at once. :)

Thanks again!

https://us-east.manta.joyent.com/arekinath/public/kpti/20180221T003551Z/platform-20180221T003551Z.tgz
https://us-east.manta.joyent.com/arekinath/public/kpti/20180221T003551Z/platform-20180221T003551Z.iso
https://us-east.manta.joyent.com/arekinath/public/kpti/20180221T003551Z/platform-20180221T003551Z.usb.bz2


Hi,

with the USB image I am unable to use virtual console both on my Toshiba 
Satellite L855 laptop (Ivy Bridge) and as well as on Linux KVM (CPU is 
set as host-passthrough). The former KPTI image's virtual console works 
in both cases.


Now I see what is my problem. When I boot this and the newest image 
(platform-20180305T101513Z), serial console in form of `os_console` 
variable is set to `ttyb`, but I don't have any serial console on the 
laptop. When I manually set it to `vga`, I am all good. (On Linux KVM 
it's probably the same thing.)


Michal



However I am able to deploy SmartOS from the older KPTI image and then 
upgrade the system with the KPTI+PCID USB image and SSH to the system. 
When on Linux KVM I was able to run many tests from Phoronix Test Suite 
(from pkgsrc) - they all passed, could not break it (sorry :) ). I'll do 
the same on bare metal this week.


Michal
*smartos-discuss* | Archives 
<https://www.listbox.com/member/archive/184463/=now> 
<https://www.listbox.com/member/archive/rss/184463/29472262-a6a0dce3> | 
Modify 
<https://www.listbox.com/member/?;> 
Your Subscription	[Powered by Listbox] <http://www.listbox.com>





---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] KPTI testing help wanted (PCID edition)

[Michal Nowak]

On 02/23/2018 03:17 AM, Alex Wilson wrote:Hi all,

Same deal as before, I would like to solicit testers for a DEBUG image
of KPTI with PCID support this time. I am particularly interested in
anyone who has a Nehalem or Westmere era Xeon to try this one out, as
well as anyone who can put significant load on it. Quite a few of the
bugs we've found so far have only been visible when all the CPUs were
kept busy at once. :)

Thanks again!

https://us-east.manta.joyent.com/arekinath/public/kpti/20180221T003551Z/platform-20180221T003551Z.tgz
https://us-east.manta.joyent.com/arekinath/public/kpti/20180221T003551Z/platform-20180221T003551Z.iso
https://us-east.manta.joyent.com/arekinath/public/kpti/20180221T003551Z/platform-20180221T003551Z.usb.bz2

Hi,
with the USB image I am unable to use virtual console both on my
Toshiba Satellite L855 laptop (Ivy Bridge) and as well as on Linux
KVM (CPU is set as host-passthrough). The former KPTI image's
virtual console works in both cases.
However I am able to deploy SmartOS from the older KPTI image and
then upgrade the system with the KPTI+PCID USB image and SSH to the
system. When on Linux KVM I was able to run many tests from Phoronix
Test Suite (from pkgsrc) - they all passed, could not break it
(sorry :) ). I'll do the same on bare metal this week.
Michal
  smartos-discuss | Archives

 | Modify
 Your Subscription

Re: [smartos-discuss] hanging boot on ASUS box

[Michal Nowak]

On Thursday, February 8, 2018 at 2:56 PM, Robert Fisher  
wrote:
> Hello all. I'm trying to put SmartOS on a reasonably modern machine with
> an
> ASUS H110 board and a 6th gen i5 CPU.
> 
> It start booting, then hangs, for ever. With -v I get to
> 
> root on /ramdisk:a fstype ufs
> 
> and no further. Cursor keeps flashing, that's all. I've tried every BIOS
> setting that I think might make any difference, and unplugged everything
> unpluggable form the main board.
> 
> I also tried a Solaris 11 image, and I get the same thing. (Solaris says
> something like "root on rpool fstype zfs" of course, but it's clearly the
> same point.)  I have definitely had that box boot Solaris, though it's
> possible the BIOS has been upgraded since then.
> 
> FreeBSD boots fine.
> 
> If anyone can think of anything worth trying, I'm all ears.
> 
> 
> Rob
> 

I suggest you to disable ACPI and/or XHCI (-B disable-xhci=true) either in BIOS 
or in bootloader. Those two sub-systems I have most problems with on newer 
Intel systems.

Michal


---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com