Re: [ceph-users] Luminous with osd flapping, slow requests when deep scrubbing

2018-10-16 Thread Christian Balzer
t: Tuesday, 16 October, 2018 08:51:36 > > Subject: Re: [ceph-users] Luminous with osd flapping, slow requests when > > deep scrubbing > > > Hello, > > > > On Mon, 15 Oct 2018 12:26:50 +0100 (BST) Andrei Mikhailovsky wrote: > > > >> Hello, > >&g

Re: [ceph-users] Luminous with osd flapping, slow requests when deep scrubbing

2018-10-16 Thread Andrei Mikhailovsky
Hi Christian, - Original Message - > From: "Christian Balzer" > To: "ceph-users" > Cc: "Andrei Mikhailovsky" > Sent: Tuesday, 16 October, 2018 08:51:36 > Subject: Re: [ceph-users] Luminous with osd flapping, slow requests when deep > sc

Re: [ceph-users] Luminous with osd flapping, slow requests when deep scrubbing

2018-10-16 Thread Christian Balzer
Hello, On Mon, 15 Oct 2018 12:26:50 +0100 (BST) Andrei Mikhailovsky wrote: > Hello, > > I am currently running Luminous 12.2.8 on Ubuntu with 4.15.0-36-generic > kernel from the official ubuntu repo. The cluster has 4 mon + osd servers. > Each osd server has the total of 9 spinning osds

Re: [ceph-users] Luminous with osd flapping, slow requests when deep scrubbing

2018-10-15 Thread Igor Fedotov
Perhaps this is the same issue as indicated here: https://tracker.ceph.com/issues/36364 Can you check OSD iostat reports for similarities to this ticket, please? Thanks, Igor On 10/15/2018 2:26 PM, Andrei Mikhailovsky wrote: Hello, I am currently running Luminous 12.2.8 on Ubuntu with

Re: [ceph-users] Luminous with osd flapping, slow requests when deep scrubbing

2018-10-15 Thread Eugen Block
Hi Andrei, we have been using the script from [1] to define the number of PGs to deep-scrub in parallel, we currently use MAXSCRUBS=4, you could start with 1 to minimize performance impacts. And these are the scrub settings from our ceph.conf: ceph:~ # grep scrub /etc/ceph/ceph.conf

[ceph-users] Luminous with osd flapping, slow requests when deep scrubbing

2018-10-15 Thread Andrei Mikhailovsky
Hello, I am currently running Luminous 12.2.8 on Ubuntu with 4.15.0-36-generic kernel from the official ubuntu repo. The cluster has 4 mon + osd servers. Each osd server has the total of 9 spinning osds and 1 ssd for the hdd and ssd pools. The hdds are backed by the S3710 ssds for journaling