On 2019-04-14 22:47, Alex McWhirter wrote:
> On 2019-04-14 17:07, Strahil Nikolov wrote:
>
>> Some kernels do not like values below 5%, thus I prefer to use
>> vm.dirty_bytes & vm.dirty_background_bytes.
>> Try the following ones (comment out the vdsm.conf values ):
>>
>> vm.dirty_background_bytes = 200000000
>> vm.dirty_bytes = 450000000
>> It's more like shooting in the dark , but it might help.
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> В неделя, 14 април 2019 г., 19:06:07 ч. Гринуич+3, Alex McWhirter
>> <[email protected]> написа:
>>
>> On 2019-04-13 03:15, Strahil wrote:
>>> Hi,
>>>
>>> What is your dirty cache settings on the gluster servers ?
>>>
>>> Best Regards,
>>> Strahil NikolovOn Apr 13, 2019 00:44, Alex McWhirter <[email protected]>
>>> wrote:
>>>>
>>>> I have 8 machines acting as gluster servers. They each have 12 drives
>>>> raid 50'd together (3 sets of 4 drives raid 5'd then 0'd together as
>>>> one).
>>>>
>>>> They connect to the compute hosts and to each other over lacp'd 10GB
>>>> connections split across two cisco nexus switched with VPC.
>>>>
>>>> Gluster has the following set.
>>>>
>>>> performance.write-behind-window-size: 4MB
>>>> performance.flush-behind: on
>>>> performance.stat-prefetch: on
>>>> server.event-threads: 4
>>>> client.event-threads: 8
>>>> performance.io-thread-count: 32
>>>> network.ping-timeout: 30
>>>> cluster.granular-entry-heal: enable
>>>> performance.strict-o-direct: on
>>>> storage.owner-gid: 36
>>>> storage.owner-uid: 36
>>>> features.shard: on
>>>> cluster.shd-wait-qlength: 10000
>>>> cluster.shd-max-threads: 8
>>>> cluster.locking-scheme: granular
>>>> cluster.data-self-heal-algorithm: full
>>>> cluster.server-quorum-type: server
>>>> cluster.quorum-type: auto
>>>> cluster.eager-lock: enable
>>>> network.remote-dio: off
>>>> performance.low-prio-threads: 32
>>>> performance.io-cache: off
>>>> performance.read-ahead: off
>>>> performance.quick-read: off
>>>> auth.allow: *
>>>> user.cifs: off
>>>> transport.address-family: inet
>>>> nfs.disable: off
>>>> performance.client-io-threads: on
>>>>
>>>>
>>>> I have the following sysctl values on gluster client and servers,
>>>> using
>>>> libgfapi, MTU 9K
>>>>
>>>> net.core.rmem_max = 134217728
>>>> net.core.wmem_max = 134217728
>>>> net.ipv4.tcp_rmem = 4096 87380 134217728
>>>> net.ipv4.tcp_wmem = 4096 65536 134217728
>>>> net.core.netdev_max_backlog = 300000
>>>> net.ipv4.tcp_moderate_rcvbuf =1
>>>> net.ipv4.tcp_no_metrics_save = 1
>>>> net.ipv4.tcp_congestion_control=htcp
>>>>
>>>> reads with this setup are perfect, benchmarked in VM to be about
>>>> 770MB/s
>>>> sequential with disk access times of < 1ms. Writes on the other hand
>>>> are
>>>> all over the place. They peak around 320MB/s sequential write, which
>>>> is
>>>> what i expect but it seems as if there is some blocking going on.
>>>>
>>>> During the write test i will hit 320MB/s briefly, then 0MB/s as disk
>>>> access time shoot to over 3000ms, then back to 320MB/s. It averages
>>>> out
>>>> to about 110MB/s afterwards.
>>>>
>>>> Gluster version is 3.12.15 ovirt is 4.2.7.5
>>>>
>>>> Any ideas on what i could tune to eliminate or minimize that blocking?
>>>> _______________________________________________
>>>> Users mailing list -- [email protected]
>>>> To unsubscribe send an email to [email protected]
>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>>> oVirt Code of Conduct:
>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>> List Archives:
>>>> https://lists.ovirt.org/archives/list/[email protected]/message/Z7F72BKYKAGICERZETSA4KCLQYR3AORR/
>>>>
>>
>>> _______________________________________________
>>> Users mailing list -- [email protected]
>>> To unsubscribe send an email to [email protected]
>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>> oVirt Code of Conduct:
>>> https://www.ovirt.org/community/about/community-guidelines/
>>> List Archives:
>>> https://lists.ovirt.org/archives/list/[email protected]/message/FMB6NCNJL2WKEDWPAM4OJIRF2GIDJUUE/
>>
>> Just the vdsm defaults
>>
>> vm.dirty_ratio = 5
>> vm.dirty_background_ratio = 2
>>
>> these boxes only have 8gb of ram as well, so those percentages should be
>> super small.
>>
>> _______________________________________________
>> Users mailing list -- [email protected]
>> To unsubscribe send an email to [email protected]
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>> https://lists.ovirt.org/archives/list/[email protected]/message/5U6QGARQSLFXMPP2EB57DSEACZ3H5SBY/
>
> i will try this,
>
> I went in and disabled TCP offload on all the nics, huge performance boost.
> went from 110MB/s to 240MB/s seq writes, reads lost a bit of performance
> going down to 680MB/s, but that's a decent trade off. Latency is still really
> high though, need to work on that. I think some more TCP tuning might help.
>
> _______________________________________________
> Users mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/[email protected]/message/XFZQSMKDZZTTPDIZ4AFA2X2TA3XWIU6Y/
Those changes didn't do a whole lot, but i ended up enabling
performance.read-ahead on the gluster volume. my blockdev read ahead
values were already 8192, which seemed good enough. Not sure if ovirt
set those, or if it's just the defaults of my raid controller.
Anyways up to 350MB/s writes, 700MB/s reads. Which so happens to
correlate with the saturation of my 10G network. Latency is still a
slight issue, but at least now im not blocking :)
_______________________________________________
Users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/[email protected]/message/5COPHAIVCVK42KMMGWZQVMNGDH6Q32ZC/
