On 2019-04-14 17:07, Strahil Nikolov wrote:
> Some kernels do not like values below 5%, thus I prefer to use
> vm.dirty_bytes & vm.dirty_background_bytes.
> Try the following ones (comment out the vdsm.conf values ):
>
> vm.dirty_background_bytes = 200000000
> vm.dirty_bytes = 450000000
> It's more like shooting in the dark , but it might help.
>
> Best Regards,
> Strahil Nikolov
>
> В неделя, 14 април 2019 г., 19:06:07 ч. Гринуич+3, Alex McWhirter
> <[email protected]> написа:
>
> On 2019-04-13 03:15, Strahil wrote:
>> Hi,
>>
>> What is your dirty cache settings on the gluster servers ?
>>
>> Best Regards,
>> Strahil NikolovOn Apr 13, 2019 00:44, Alex McWhirter <[email protected]>
>> wrote:
>>>
>>> I have 8 machines acting as gluster servers. They each have 12 drives
>>> raid 50'd together (3 sets of 4 drives raid 5'd then 0'd together as
>>> one).
>>>
>>> They connect to the compute hosts and to each other over lacp'd 10GB
>>> connections split across two cisco nexus switched with VPC.
>>>
>>> Gluster has the following set.
>>>
>>> performance.write-behind-window-size: 4MB
>>> performance.flush-behind: on
>>> performance.stat-prefetch: on
>>> server.event-threads: 4
>>> client.event-threads: 8
>>> performance.io-thread-count: 32
>>> network.ping-timeout: 30
>>> cluster.granular-entry-heal: enable
>>> performance.strict-o-direct: on
>>> storage.owner-gid: 36
>>> storage.owner-uid: 36
>>> features.shard: on
>>> cluster.shd-wait-qlength: 10000
>>> cluster.shd-max-threads: 8
>>> cluster.locking-scheme: granular
>>> cluster.data-self-heal-algorithm: full
>>> cluster.server-quorum-type: server
>>> cluster.quorum-type: auto
>>> cluster.eager-lock: enable
>>> network.remote-dio: off
>>> performance.low-prio-threads: 32
>>> performance.io-cache: off
>>> performance.read-ahead: off
>>> performance.quick-read: off
>>> auth.allow: *
>>> user.cifs: off
>>> transport.address-family: inet
>>> nfs.disable: off
>>> performance.client-io-threads: on
>>>
>>>
>>> I have the following sysctl values on gluster client and servers,
>>> using
>>> libgfapi, MTU 9K
>>>
>>> net.core.rmem_max = 134217728
>>> net.core.wmem_max = 134217728
>>> net.ipv4.tcp_rmem = 4096 87380 134217728
>>> net.ipv4.tcp_wmem = 4096 65536 134217728
>>> net.core.netdev_max_backlog = 300000
>>> net.ipv4.tcp_moderate_rcvbuf =1
>>> net.ipv4.tcp_no_metrics_save = 1
>>> net.ipv4.tcp_congestion_control=htcp
>>>
>>> reads with this setup are perfect, benchmarked in VM to be about
>>> 770MB/s
>>> sequential with disk access times of < 1ms. Writes on the other hand
>>> are
>>> all over the place. They peak around 320MB/s sequential write, which
>>> is
>>> what i expect but it seems as if there is some blocking going on.
>>>
>>> During the write test i will hit 320MB/s briefly, then 0MB/s as disk
>>> access time shoot to over 3000ms, then back to 320MB/s. It averages
>>> out
>>> to about 110MB/s afterwards.
>>>
>>> Gluster version is 3.12.15 ovirt is 4.2.7.5
>>>
>>> Any ideas on what i could tune to eliminate or minimize that blocking?
>>> _______________________________________________
>>> Users mailing list -- [email protected]
>>> To unsubscribe send an email to [email protected]
>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>> oVirt Code of Conduct:
>>> https://www.ovirt.org/community/about/community-guidelines/
>>> List Archives:
>>> https://lists.ovirt.org/archives/list/[email protected]/message/Z7F72BKYKAGICERZETSA4KCLQYR3AORR/
>>>
>
>> _______________________________________________
>> Users mailing list -- [email protected]
>> To unsubscribe send an email to [email protected]
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>> https://lists.ovirt.org/archives/list/[email protected]/message/FMB6NCNJL2WKEDWPAM4OJIRF2GIDJUUE/
>
> Just the vdsm defaults
>
> vm.dirty_ratio = 5
> vm.dirty_background_ratio = 2
>
> these boxes only have 8gb of ram as well, so those percentages should be
> super small.
>
> _______________________________________________
> Users mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/[email protected]/message/5U6QGARQSLFXMPP2EB57DSEACZ3H5SBY/
i will try this,
I went in and disabled TCP offload on all the nics, huge performance
boost. went from 110MB/s to 240MB/s seq writes, reads lost a bit of
performance going down to 680MB/s, but that's a decent trade off.
Latency is still really high though, need to work on that. I think some
more TCP tuning might help.
_______________________________________________
Users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/[email protected]/message/XFZQSMKDZZTTPDIZ4AFA2X2TA3XWIU6Y/
