> On 25 Jul 2018, at 22:42, Michal Nowak <mno...@startmail.com> wrote:
>
> On 07/25/18 08:49 PM, Len Weincier wrote:
>> Hi
>> I see this commit and the hosts where we see issues have 2 NVMe devs used as
>> slogs if that helps
>> https://github.com/joyent/illumos-joyent/commit/f78cdc34af236a6199dd9e21376f4a46348c0d56
>> Len
>>> On 25 Jul 2018, at 16:58, Len Weincier <l...@cloudafrica.net
>>> <mailto:l...@cloudafrica.net>> wrote:
>>>
>>> Hi
>>>
>>> We a very strange situation trying to upgrade to a newer smartos image
>>> where the disk I/O is *very* slow.
>>>
>>> I have been working through the released images and the last one that works
>>> 100% is 20180329T002644Z
>>>
>>> From 20180412T003259Z onwards, the release with the new zfs features like
>>> spacemaps etc, the hosts become unusable in terms of disk i/o
>>>
>>> In our testing with the lab machine with only 128G ram we see no
>>> pathologies.
>>>
>>> Hosts are running ALL SSDs (RAIDZ2), and Intel Gold 6150 x2 processors on
>>> an SMC X11DPH-T board..
>>> The lab machine with 128GB RAM has exactly the same processors, board, and
>>> SSD-only setup - except for RAM..
>>>
>>> On a production machine with 768G ram and the newer image for eg zfs create
>>> -V 10G zones/test takes 2 minutes while at the same time iostat is showing
>>> the disks as relatively idle (%b = 10)
>>>
>>> For example inside an ubuntu kvm with postgres we are seeing 40% wait time
>>> for any disk i/o and there are only 2 vm's running, underlying disks
>>> essentially idle.
>>>
>>> Is there anything we can look at to get to the bottom of this as it pretty
>>> critical and affecting our customers
>>>
>>> Any help appreciated
>>> Thanks
>>> Len
>
> The log
> (https://us-east.manta.joyent.com/Joyent_Dev/public/SmartOS/smartos.html#20180412T003259Z)
> also says "Implement KPTI". Does the problem go away when you boot with
> kpti=0? I've seen disk IO going haywire for very old CPU without
> process-context identifiers (PCID).
>
> Michal
>
Those are the new Intel Gold 6150 CPU’s, but I can try that
Len
-------------------------------------------
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: https://www.listbox.com
