Re: [ceph-users] osd_disk_thread_ioprio_priority help

2017-03-15 Thread Florian Haas 
On Wed, Mar 15, 2017 at 2:41 AM, Alex Gorbachev  
wrote:
> On Mon, Mar 13, 2017 at 6:09 AM, Florian Haas  wrote:
>> On Mon, Mar 13, 2017 at 11:00 AM, Dan van der Ster  
>> wrote:
 I'm sorry, I may have worded that in a manner that's easy to
 misunderstand. I generally *never* suggest that people use CFQ on
 reasonably decent I/O hardware, and thus have never come across any
 need to set this specific ceph.conf parameter.
>>>
>>> OTOH, cfq *does* help our hammer clusters. deadline's default
>>> behaviour is to delay writes up to 5 seconds if the disk is busy
>>> reading -- which it is, of couse, while deep scrubbing. And deadline
>>> does not offer any sort of fairness between processes accessing the
>>> same disk (which is admittedly less of an issue in jewel). But back in
>>> hammer days it was nice to be able to make the disk threads only read
>>> while the disk was otherwise idle.
>>
>> Thanks for pointing out the default 5000-ms write deadline. We
>> frequently tune that down to 1500ms. Disabling front merges also
>> sometimes seems to help.
>>
>> For the archives: those settings are in
>> /sys/block/*/queue/iosched/{write_expire,front_merges} and can be
>> persisted on Debian/Ubuntu with sysfsutils.
>
> Hey Florian :).

Привет Лёша! Long time no talk. :)

> I wrote a quick udev rule to do this on Ubuntu, here is is for others'
> reference.  Also saw earlier a recommendation to increase
> read_ahead_kb to 4096 for slower spinning disks
>
> root@croc1:/etc/udev/rules.d# cat 99-storcium-hdd.rules
>
> # Set write deadline to 1500 and disable front merges, and increase
> read_ahead_kb to 4096
> ACTION=="add|change", KERNEL=="sd*", SUBSYSTEM=="block",
> ATTR{queue/iosched/write_expire}="1500",
> ATTR{queue/iosched/front_merges}="0", ATTR{queue/read_ahead_kb}="4096"

Sure, that's an entirely viable alternative to using sysfsutils.
Indeed, rather a more elegant and contemporary one. :)

Cheers,
Florian
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd_disk_thread_ioprio_priority help

2017-03-14 Thread Alex Gorbachev 
On Mon, Mar 13, 2017 at 6:09 AM, Florian Haas  wrote:
> On Mon, Mar 13, 2017 at 11:00 AM, Dan van der Ster  
> wrote:
>>> I'm sorry, I may have worded that in a manner that's easy to
>>> misunderstand. I generally *never* suggest that people use CFQ on
>>> reasonably decent I/O hardware, and thus have never come across any
>>> need to set this specific ceph.conf parameter.
>>
>> OTOH, cfq *does* help our hammer clusters. deadline's default
>> behaviour is to delay writes up to 5 seconds if the disk is busy
>> reading -- which it is, of couse, while deep scrubbing. And deadline
>> does not offer any sort of fairness between processes accessing the
>> same disk (which is admittedly less of an issue in jewel). But back in
>> hammer days it was nice to be able to make the disk threads only read
>> while the disk was otherwise idle.
>
> Thanks for pointing out the default 5000-ms write deadline. We
> frequently tune that down to 1500ms. Disabling front merges also
> sometimes seems to help.
>
> For the archives: those settings are in
> /sys/block/*/queue/iosched/{write_expire,front_merges} and can be
> persisted on Debian/Ubuntu with sysfsutils.

Hey Florian :).

I wrote a quick udev rule to do this on Ubuntu, here is is for others'
reference.  Also saw earlier a recommendation to increase
read_ahead_kb to 4096 for slower spinning disks

root@croc1:/etc/udev/rules.d# cat 99-storcium-hdd.rules

# Set write deadline to 1500 and disable front merges, and increase
read_ahead_kb to 4096
ACTION=="add|change", KERNEL=="sd*", SUBSYSTEM=="block",
ATTR{queue/iosched/write_expire}="1500",
ATTR{queue/iosched/front_merges}="0", ATTR{queue/read_ahead_kb}="4096"

Best regards,
Alex

>
> Cheers,
> Florian
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd_disk_thread_ioprio_priority help

2017-03-13 Thread Nick Fisk 
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Florian Haas
> Sent: 13 March 2017 10:09
> To: Dan van der Ster <d...@vanderster.com>
> Cc: ceph-users <ceph-users@lists.ceph.com>
> Subject: Re: [ceph-users] osd_disk_thread_ioprio_priority help
> 
> On Mon, Mar 13, 2017 at 11:00 AM, Dan van der Ster
> <d...@vanderster.com> wrote:
> >> I'm sorry, I may have worded that in a manner that's easy to
> >> misunderstand. I generally *never* suggest that people use CFQ on
> >> reasonably decent I/O hardware, and thus have never come across any
> >> need to set this specific ceph.conf parameter.
> >
> > OTOH, cfq *does* help our hammer clusters. deadline's default
> > behaviour is to delay writes up to 5 seconds if the disk is busy
> > reading -- which it is, of couse, while deep scrubbing. And deadline
> > does not offer any sort of fairness between processes accessing the
> > same disk (which is admittedly less of an issue in jewel). But back in
> > hammer days it was nice to be able to make the disk threads only read
> > while the disk was otherwise idle.
> 
> Thanks for pointing out the default 5000-ms write deadline. We frequently
> tune that down to 1500ms. Disabling front merges also sometimes seems to
> help.
> 
> For the archives: those settings are in
> /sys/block/*/queue/iosched/{write_expire,front_merges} and can be
> persisted on Debian/Ubuntu with sysfsutils.

Also it may be of some interest that in Linux 4.10 there is new background
priority writeback functionality

https://kernelnewbies.org/Linux_4.10#head-f6ecae920c0660b7f4bcee913f2c71a859
dcc184

I've found this makes quite a big difference to read latency if the cluster
is under a heavy writes and the WBthrottle allows 5000 IO's to queue up by
default.

> 
> Cheers,
> Florian
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd_disk_thread_ioprio_priority help

2017-03-13 Thread Florian Haas 
On Mon, Mar 13, 2017 at 11:00 AM, Dan van der Ster  wrote:
>> I'm sorry, I may have worded that in a manner that's easy to
>> misunderstand. I generally *never* suggest that people use CFQ on
>> reasonably decent I/O hardware, and thus have never come across any
>> need to set this specific ceph.conf parameter.
>
> OTOH, cfq *does* help our hammer clusters. deadline's default
> behaviour is to delay writes up to 5 seconds if the disk is busy
> reading -- which it is, of couse, while deep scrubbing. And deadline
> does not offer any sort of fairness between processes accessing the
> same disk (which is admittedly less of an issue in jewel). But back in
> hammer days it was nice to be able to make the disk threads only read
> while the disk was otherwise idle.

Thanks for pointing out the default 5000-ms write deadline. We
frequently tune that down to 1500ms. Disabling front merges also
sometimes seems to help.

For the archives: those settings are in
/sys/block/*/queue/iosched/{write_expire,front_merges} and can be
persisted on Debian/Ubuntu with sysfsutils.

Cheers,
Florian
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd_disk_thread_ioprio_priority help

2017-03-13 Thread Dan van der Ster 
On Mon, Mar 13, 2017 at 10:35 AM, Florian Haas  wrote:
> On Sun, Mar 12, 2017 at 9:07 PM, Laszlo Budai  wrote:
>> Hi Florian,
>>
>> thank you for your answer.
>>
>> We have already set the IO scheduler to cfq in order to be able to lower the
>> priority of the scrub operations.
>> My problem is that I've found different values set for the same parameter,
>> and in each case they were doing it in order to achieve the same thing as we
>> do.
>> This is why I was asking for some more details about this parameter. Is the
>> value 0 or 7 which will set the scrub to the lowest priority? I want my ceph
>> cluster to be responsive for the client requests, and do the scrub in the
>> background.
>>
>> I'm open to any ideas/suggestions which would help to improove the cluster's
>> responsiveness during deep scrub operations.
>
> I'm sorry, I may have worded that in a manner that's easy to
> misunderstand. I generally *never* suggest that people use CFQ on
> reasonably decent I/O hardware, and thus have never come across any
> need to set this specific ceph.conf parameter.

OTOH, cfq *does* help our hammer clusters. deadline's default
behaviour is to delay writes up to 5 seconds if the disk is busy
reading -- which it is, of couse, while deep scrubbing. And deadline
does not offer any sort of fairness between processes accessing the
same disk (which is admittedly less of an issue in jewel). But back in
hammer days it was nice to be able to make the disk threads only read
while the disk was otherwise idle.

-- Dan

>
> Maybe if you share your full ceph.conf and hardware details, I or
> others on this list can offer more useful suggestions than tweaking
> osd_disk_thread_ioprio_priority.
>
> Cheers,
> Florian
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd_disk_thread_ioprio_priority help

2017-03-13 Thread Florian Haas 
On Sun, Mar 12, 2017 at 9:07 PM, Laszlo Budai  wrote:
> Hi Florian,
>
> thank you for your answer.
>
> We have already set the IO scheduler to cfq in order to be able to lower the
> priority of the scrub operations.
> My problem is that I've found different values set for the same parameter,
> and in each case they were doing it in order to achieve the same thing as we
> do.
> This is why I was asking for some more details about this parameter. Is the
> value 0 or 7 which will set the scrub to the lowest priority? I want my ceph
> cluster to be responsive for the client requests, and do the scrub in the
> background.
>
> I'm open to any ideas/suggestions which would help to improove the cluster's
> responsiveness during deep scrub operations.

I'm sorry, I may have worded that in a manner that's easy to
misunderstand. I generally *never* suggest that people use CFQ on
reasonably decent I/O hardware, and thus have never come across any
need to set this specific ceph.conf parameter.

Maybe if you share your full ceph.conf and hardware details, I or
others on this list can offer more useful suggestions than tweaking
osd_disk_thread_ioprio_priority.

Cheers,
Florian
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd_disk_thread_ioprio_priority help

2017-03-12 Thread Florian Haas 
On Sat, Mar 11, 2017 at 4:24 PM, Laszlo Budai  wrote:
>>> Can someone explain the meaning of osd_disk_thread_ioprio_priority. I'm
>>> [...]
>>>
>>> Now I am confused  :(
>>>
>>> Can somebody bring some light here?
>>
>>
>> Only to confuse you some more. If you are running Jewel or above then
>> scrubbing is now done in the main operation thread and so setting this
>> value
>> will have no effect.
>
>
> There is the hammer version of ceph.

As the documentation
(http://docs.ceph.com/docs/hammer/rados/configuration/osd-config-ref/)
explains, osd_disk_thread_ioprio_priority is ineffective unless you're
using the CFQ I/O scheduler. Most server I/O stack configurations
(including typical Ceph OSD nodes) run with the deadline scheduler
either by default or by best-practice configuration, so this option
has no effect on such systems.

You can check /sys/block//queue/scheduler to see which
scheduler your devices are currently using.

Cheers,
Florian
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd_disk_thread_ioprio_priority help

2017-03-11 Thread Laszlo Budai 


On 11.03.2017 16:25, Nick Fisk wrote:

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
Laszlo Budai
Sent: 11 March 2017 13:51
To: ceph-users <ceph-users@lists.ceph.com>
Subject: [ceph-users] osd_disk_thread_ioprio_priority help

Hello,


Can someone explain the meaning of osd_disk_thread_ioprio_priority. I'm
reading the definition from this page:
https://access.redhat.com/documentation/en-
us/red_hat_ceph_storage/1.3/html/configuration_guide/osd_configuration_
reference
it says: "It sets the ioprio_set(2) I/O scheduling priority of the disk

thread

ranging from 0 (highest) to 7 (lowest)." What is the so called disk

thread?


Then I found documents using both extreme values (0 and 7) for achieving
the same stuff make the scrubbing low priority.
https://ceph.com/geen-categorie/ceph-reduce-osd-scrub-priority/ and
http://dachary.org/?p=3268 are using the value 7 for reducing the priority

of

scrub, while here: http://ceph-users.ceph.narkive.com/AMMP3r5s/osd-
scrub-sleep-osd-scrub-chunk-min-max and
https://indico.cern.ch/event/588794/contributions/2374222/attachments/13
83112/2103509/Configuring_Ceph.pdf the value 0 is used.

Now I am confused  :(

Can somebody bring some light here?


Only to confuse you some more. If you are running Jewel or above then
scrubbing is now done in the main operation thread and so setting this value
will have no effect.


There is the hammer version of ceph.





Thank you,
Laszlo

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd_disk_thread_ioprio_priority help

2017-03-11 Thread Nick Fisk 
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Laszlo Budai
> Sent: 11 March 2017 13:51
> To: ceph-users <ceph-users@lists.ceph.com>
> Subject: [ceph-users] osd_disk_thread_ioprio_priority help
> 
> Hello,
> 
> 
> Can someone explain the meaning of osd_disk_thread_ioprio_priority. I'm
> reading the definition from this page:
> https://access.redhat.com/documentation/en-
> us/red_hat_ceph_storage/1.3/html/configuration_guide/osd_configuration_
> reference
> it says: "It sets the ioprio_set(2) I/O scheduling priority of the disk
thread
> ranging from 0 (highest) to 7 (lowest)." What is the so called disk
thread?
> 
> Then I found documents using both extreme values (0 and 7) for achieving
> the same stuff make the scrubbing low priority.
> https://ceph.com/geen-categorie/ceph-reduce-osd-scrub-priority/ and
> http://dachary.org/?p=3268 are using the value 7 for reducing the priority
of
> scrub, while here: http://ceph-users.ceph.narkive.com/AMMP3r5s/osd-
> scrub-sleep-osd-scrub-chunk-min-max and
> https://indico.cern.ch/event/588794/contributions/2374222/attachments/13
> 83112/2103509/Configuring_Ceph.pdf the value 0 is used.
> 
> Now I am confused  :(
> 
> Can somebody bring some light here?

Only to confuse you some more. If you are running Jewel or above then
scrubbing is now done in the main operation thread and so setting this value
will have no effect.

> 
> Thank you,
> Laszlo
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] osd_disk_thread_ioprio_priority help

2017-03-11 Thread Laszlo Budai 

Hello,


Can someone explain the meaning of osd_disk_thread_ioprio_priority. I'm reading 
the definition from this page: 
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.3/html/configuration_guide/osd_configuration_reference
it says: "It sets the ioprio_set(2) I/O scheduling priority of the disk thread 
ranging from 0 (highest) to 7 (lowest)." What is the so called disk thread?

Then I found documents using both extreme values (0 and 7) for achieving the 
same stuff make the scrubbing low priority.
https://ceph.com/geen-categorie/ceph-reduce-osd-scrub-priority/ and 
http://dachary.org/?p=3268 are using the value 7 for reducing the priority of 
scrub, while here: 
http://ceph-users.ceph.narkive.com/AMMP3r5s/osd-scrub-sleep-osd-scrub-chunk-min-max
 and 
https://indico.cern.ch/event/588794/contributions/2374222/attachments/1383112/2103509/Configuring_Ceph.pdf
 the value 0 is used.

Now I am confused  :(

Can somebody bring some light here?

Thank you,
Laszlo

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com