[ceph-users] Re: 6 pgs not deep-scrubbed in time

2024-01-30 Thread Michel Niyoyita 
Dear team,

below is the output of ceph df command and the ceph version I am running

 ceph df
--- RAW STORAGE ---
CLASS SIZEAVAIL USED  RAW USED  %RAW USED
hdd433 TiB  282 TiB  151 TiB   151 TiB  34.82
TOTAL  433 TiB  282 TiB  151 TiB   151 TiB  34.82

--- POOLS ---
POOL   ID  PGS   STORED  OBJECTS USED  %USED  MAX AVAIL
device_health_metrics   11  1.1 MiB3  3.2 MiB  0 73 TiB
.rgw.root   2   32  3.7 KiB8   96 KiB  0 73 TiB
default.rgw.log 3   32  3.6 KiB  209  408 KiB  0 73 TiB
default.rgw.control 4   32  0 B8  0 B  0 73 TiB
default.rgw.meta5   32382 B2   24 KiB  0 73 TiB
volumes 6  128   21 TiB5.68M   62 TiB  22.09 73 TiB
images  7   32  878 GiB  112.50k  2.6 TiB   1.17 73 TiB
backups 8   32  0 B0  0 B  0 73 TiB
vms 9   32  881 GiB  174.30k  2.5 TiB   1.13 73 TiB
testbench  10   32  0 B0  0 B  0 73 TiB
root@ceph-mon1:~# ceph --version
ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific
(stable)
root@ceph-mon1:~#

please advise accordingly

Michel

On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder  wrote:

> You will have to look at the output of "ceph df" and make a decision to
> balance "objects per PG" and "GB per PG". Increase he PG count for the
> pools with the worst of these two numbers most such that it balances out as
> much as possible. If you have pools that see significantly more user-IO
> than others, prioritise these.
>
> You will have to find out for your specific cluster, we can only give
> general guidelines. Make changes, run benchmarks, re-evaluate. Take the
> time for it. The better you know your cluster and your users, the better
> the end result will be.
>
> Best regards,
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> 
> From: Michel Niyoyita 
> Sent: Monday, January 29, 2024 2:04 PM
> To: Janne Johansson
> Cc: Frank Schilder; E Taka; ceph-users
> Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time
>
> This is how it is set , if you suggest to make some changes please advises.
>
> Thank you.
>
>
> ceph osd pool ls detail
> pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 1407
> flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application
> mgr_devicehealth
> pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 flags
> hashpspool stripe_width 0 application rgw
> pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
> 1394 flags hashpspool stripe_width 0 application rgw
> pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
> 1395 flags hashpspool stripe_width 0 application rgw
> pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
> 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw
> pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 108802 lfor
> 0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
> removed_snaps_queue
> [22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c]
> pool 7 'images' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 flags
> hashpspool,selfmanaged_snaps stripe_width 0 application rbd
> pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 flags
> hashpspool stripe_width 0 application rbd
> pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 lfor
> 0/561/559 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
> removed_snaps_queue [3fa~1,3fc~3,400~1,402~1]
> pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 lfor
> 0/20931/20929 flags hashpspool stripe_width 0
>
>
> On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita  mico...@gmail.com>> wrote:
> Thank you Janne ,
>
> no need of setting some flags like ceph osd set nodeep-scrub  ???
>
> Thank you
>
> On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson

[ceph-users] NFS HA - "virtual_ip": null after upgrade to reef

2024-01-30 Thread Torkil Svensgaard 

Hi

Last week we created an NFS service like this:

"
ceph nfs cluster create jumbo "ceph-flash1,ceph-flash2,ceph-flash3" 
--ingress --virtual_ip 172.21.15.74/22 --ingress-mode keepalive-only

"

Worked like a charm. Yesterday we upgraded from 17.2.7 to 18.20.0 and 
the NFS virtual IP seems to have gone missing in the process:


"
# ceph nfs cluster info jumbo
{
  "jumbo": {
"backend": [
  {
"hostname": "ceph-flash1",
"ip": "172.21.15.148",
"port": 2049
  }
],
"virtual_ip": null
  }
}
"

Service spec:

"
service_type: nfs
service_id: jumbo
service_name: nfs.jumbo
placement:
  count: 1
  hosts:
  - ceph-flash1
  - ceph-flash2
  - ceph-flash3
spec:
  port: 2049
  virtual_ip: 172.21.15.74
"

I've tried restarting the nfs.jumbo service which didn't help. Suggestions?

Mvh.

Torkil

--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: tor...@drcmr.dk
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] how to avoid pglogs dups bug in Pacific

2024-01-30 Thread ADRIAN NICOLAE 

 Hi,

 I'm running Pacific 16.2.4 and I want to start a manual pg split 
process on the data pool (from 2048 to 4096).  I'm reluctant to upgrade 
to 16.2.14/15 at this point. Can I avoid the dups bug 
(https://tracker.ceph.com/issues/53729)  if I will increase the pgs 
slowly with 32 or 64pgs at every increment instead of moving directly to 
4096 ?  I don't have the autoscaler enabled.


 Thanks.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing?

2024-01-30 Thread Jan Marek 
Hello again,

I'm sorry, I forgot attach file... :-(

Sincerely
Jan

Dne Út, led 30, 2024 at 11:09:44 CET napsal(a) Jan Marek:
> Hello Sridhar,
> 
> at Saturday I've finished upgrade proces to 18.2.1.
> 
> Cluster is now in HEALTH_OK state and performs well.
> 
> According to my colleagues there are lower latences and good
> throughput.
> 
> On OSD nodes there is relative low I/O activity.
> 
> I still have mClock profile "high_client_ops".
> 
> When I was stucked in the upgrade process, I had in logs so many
> records, see attached file. Since upgrade is complete, this
> messages went away... Can be this reason of poor
> performance?
> 
> Sincerely
> Jan Marek
> 
> Dne Čt, led 25, 2024 at 02:31:41 CET napsal(a) Jan Marek:
> > Hello Sridhar,
> > 
> > Dne Čt, led 25, 2024 at 09:53:26 CET napsal(a) Sridhar Seshasayee:
> > > Hello Jan,
> > > 
> > > Meaning of my previous post was, that CEPH cluster didn't fulfill
> > > my needs and, although I had set mClock profile to
> > > "high_client_ops" (because I have a plenty of time to rebalancing
> > > and scrubbing), my clients went to problems.
> > > 
> > > As far as the question around mClock is concerned, there are further
> > > improvements in the works to handle QoS between client ops and
> > > background scrub ops. This should help address the issue you are
> > > currently facing. See PR: https://github.com/ceph/ceph/pull/51171
> > > for more information.
> > > Also, it would be helpful to know the Ceph version you are currently 
> > > using.
> > 
> > thanks for your reply.
> > 
> > I've just in process upgrade between 17.2.6 and 18.2.1 (you can
> > see my previous posts about stuck in upgrade to reef).
> > 
> > Maybe this was cause of my problem...
> > 
> > Now I've tried give rest to the cluster to do some "background"
> > tasks (and it seems, that this was correct, because on my hosts
> > there is around 50-100MBps read and cca 10-50MBps write traffic -
> > cca 1/4-1/2 of previous load).
> > 
> > At Saturday I will change some settings on networking and I will
> > try to start upgrade process, maybe with --limit=1, to be "soft"
> > for cluster and for our clients...
> > 
> > > -Sridhar
> > 
> > Sincerely
> > Jan Marek
> > -- 
> > Ing. Jan Marek
> > University of South Bohemia
> > Academic Computer Centre
> > Phone: +420389032080
> > http://www.gnu.org/philosophy/no-word-attachments.cs.html
> 
> 
> 
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> 
> -- 
> Ing. Jan Marek
> University of South Bohemia
> Academic Computer Centre
> Phone: +420389032080
> http://www.gnu.org/philosophy/no-word-attachments.cs.html



> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io


-- 
Ing. Jan Marek
University of South Bohemia
Academic Computer Centre
Phone: +420389032080
http://www.gnu.org/philosophy/no-word-attachments.cs.html
2024-01-27T18:46:25.152544+01:00 osd1 ceph-osd[1128137]: log_channel(cluster) 
log [WRN] : failed to encode map e15773745 with expected crc
2024-01-27T18:46:25.154758+01:00 osd1 ceph-osd[1115858]: log_channel(cluster) 
log [WRN] : failed to encode map e15773745 with expected crc
2024-01-27T18:49:00.248561+01:00 osd1 ceph-osd[1115858]: log_channel(cluster) 
log [WRN] : failed to encode map e15773746 with expected crc
2024-01-27T18:49:02.294944+01:00 osd1 ceph-osd[1128137]: log_channel(cluster) 
log [WRN] : failed to encode map e15773748 with expected crc
2024-01-27T18:49:03.281543+01:00 osd1 ceph-osd[1115858]: log_channel(cluster) 
log [WRN] : failed to encode map e15773749 with expected crc
2024-01-27T18:49:03.281894+01:00 osd1 ceph-osd[1128137]: log_channel(cluster) 
log [WRN] : failed to encode map e15773749 with expected crc
2024-01-27T18:49:06.611229+01:00 osd1 ceph-osd[1115858]: log_channel(cluster) 
log [WRN] : failed to encode map e15773752 with expected crc
2024-01-27T18:49:06.612924+01:00 osd1 ceph-osd[1128137]: log_channel(cluster) 
log [WRN] : failed to encode map e15773752 with expected crc
2024-01-27T18:49:07.634638+01:00 osd1 ceph-osd[1128137]: log_channel(cluster) 
log [WRN] : failed to encode map e15773753 with expected crc
2024-01-27T18:49:08.635094+01:00 osd1 ceph-osd[1115858]: log_channel(cluster) 
log [WRN] : failed to encode map e15773754 with expected crc
2024-01-27T18:49:11.698700+01:00 osd1 ceph-osd[1115858]: log_channel(cluster) 
log [WRN] : failed to encode map e15773757 with expected crc
2024-01-27T18:49:12.935850+01:00 osd1 ceph-osd[1128137]: log_channel(cluster) 
log [WRN] : failed to encode map e15773758 with expected crc
2024-01-27T18:49:18.050429+01:00 osd1 ceph-osd[1128137]: log_channel(cluster) 
log [WRN] : failed to encode map e15773761 with expected crc
2024-01-27T18:49:20.072799+01:00 osd1 ceph-osd[1128137]: log_channel(cluster) 
log [WRN] : failed to encode map e15773763 with expected crc

[ceph-users] Changing A Ceph Cluster's Front- And/Or Back-End Networks IP Address(es)

2024-01-30 Thread duluxoz 

Hi All,

Quick Q: How easy/hard is it to change the IP networks of:

1) A Ceph Cluster's "Front-End" Network?

2) A Ceph Cluster's "Back-End" Network?

Is it a "simply" matter of:

a) Placing the Nodes in maintenance mode

b) Changing a config file (I assume it's /etc/ceph/ceph.conf) on each Node

c) Rebooting the Nodes

d) Taking each Node out of Maintenance Mode

Thanks in advance

Cheers

Dulux-Oz
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing?

2024-01-30 Thread Josh Baergen 
Ah, yeah, you hit https://tracker.ceph.com/issues/63389 during the upgrade.

Josh

On Tue, Jan 30, 2024 at 3:17 AM Jan Marek  wrote:
>
> Hello again,
>
> I'm sorry, I forgot attach file... :-(
>
> Sincerely
> Jan
>
> Dne Út, led 30, 2024 at 11:09:44 CET napsal(a) Jan Marek:
> > Hello Sridhar,
> >
> > at Saturday I've finished upgrade proces to 18.2.1.
> >
> > Cluster is now in HEALTH_OK state and performs well.
> >
> > According to my colleagues there are lower latences and good
> > throughput.
> >
> > On OSD nodes there is relative low I/O activity.
> >
> > I still have mClock profile "high_client_ops".
> >
> > When I was stucked in the upgrade process, I had in logs so many
> > records, see attached file. Since upgrade is complete, this
> > messages went away... Can be this reason of poor
> > performance?
> >
> > Sincerely
> > Jan Marek
> >
> > Dne Čt, led 25, 2024 at 02:31:41 CET napsal(a) Jan Marek:
> > > Hello Sridhar,
> > >
> > > Dne Čt, led 25, 2024 at 09:53:26 CET napsal(a) Sridhar Seshasayee:
> > > > Hello Jan,
> > > >
> > > > Meaning of my previous post was, that CEPH cluster didn't fulfill
> > > > my needs and, although I had set mClock profile to
> > > > "high_client_ops" (because I have a plenty of time to rebalancing
> > > > and scrubbing), my clients went to problems.
> > > >
> > > > As far as the question around mClock is concerned, there are further
> > > > improvements in the works to handle QoS between client ops and
> > > > background scrub ops. This should help address the issue you are
> > > > currently facing. See PR: https://github.com/ceph/ceph/pull/51171
> > > > for more information.
> > > > Also, it would be helpful to know the Ceph version you are currently 
> > > > using.
> > >
> > > thanks for your reply.
> > >
> > > I've just in process upgrade between 17.2.6 and 18.2.1 (you can
> > > see my previous posts about stuck in upgrade to reef).
> > >
> > > Maybe this was cause of my problem...
> > >
> > > Now I've tried give rest to the cluster to do some "background"
> > > tasks (and it seems, that this was correct, because on my hosts
> > > there is around 50-100MBps read and cca 10-50MBps write traffic -
> > > cca 1/4-1/2 of previous load).
> > >
> > > At Saturday I will change some settings on networking and I will
> > > try to start upgrade process, maybe with --limit=1, to be "soft"
> > > for cluster and for our clients...
> > >
> > > > -Sridhar
> > >
> > > Sincerely
> > > Jan Marek
> > > --
> > > Ing. Jan Marek
> > > University of South Bohemia
> > > Academic Computer Centre
> > > Phone: +420389032080
> > > http://www.gnu.org/philosophy/no-word-attachments.cs.html
> >
> >
> >
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> >
> > --
> > Ing. Jan Marek
> > University of South Bohemia
> > Academic Computer Centre
> > Phone: +420389032080
> > http://www.gnu.org/philosophy/no-word-attachments.cs.html
>
>
>
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> --
> Ing. Jan Marek
> University of South Bohemia
> Academic Computer Centre
> Phone: +420389032080
> http://www.gnu.org/philosophy/no-word-attachments.cs.html
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

2024-01-30 Thread Michel Niyoyita 
I tried that on one of my pool (pool id 3) but the number of pgs not
deep-scrubbed in time increased also from 55 to 100 but the number of PGs
was increased. I set also autoscale to off mode. before continue to other
pools would like to ask if so far there is no negative impact.

ceph -s
  cluster:
id: cb0caedc-eb5b-42d1-a34f-96facfda8c27
health: HEALTH_WARN
100 pgs not deep-scrubbed in time

  services:
mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M)
mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1
osd: 48 osds: 48 up (since 11M), 48 in (since 12M)
rgw: 6 daemons active (6 hosts, 1 zones)

  data:
pools:   10 pools, 609 pgs
objects: 6.03M objects, 23 TiB
usage:   151 TiB used, 282 TiB / 433 TiB avail
pgs: 603 active+clean
 4   active+clean+scrubbing+deep
 2   active+clean+scrubbing

  io:
client:   96 MiB/s rd, 573 MiB/s wr, 576 op/s rd, 648 op/s wr

root@ceph-osd3:/var/log# ceph df
--- RAW STORAGE ---
CLASS SIZEAVAIL USED  RAW USED  %RAW USED
hdd433 TiB  282 TiB  151 TiB   151 TiB  34.93
TOTAL  433 TiB  282 TiB  151 TiB   151 TiB  34.93

--- POOLS ---
POOL   ID  PGS   STORED  OBJECTS USED  %USED  MAX AVAIL
device_health_metrics   11  1.1 MiB3  3.2 MiB  0 72 TiB
.rgw.root   2   32  3.7 KiB8   96 KiB  0 72 TiB
default.rgw.log 3  256  3.6 KiB  204  408 KiB  0 72 TiB
default.rgw.control 4   32  0 B8  0 B  0 72 TiB
default.rgw.meta5   32382 B2   24 KiB  0 72 TiB
volumes 6  128   21 TiB5.74M   62 TiB  22.30 72 TiB
images  7   32  878 GiB  112.50k  2.6 TiB   1.17 72 TiB
backups 8   32  0 B0  0 B  0 72 TiB
vms 9   32  870 GiB  170.73k  2.5 TiB   1.13 72 TiB
testbench  10   32  0 B0  0 B  0 72 TiB

On Tue, Jan 30, 2024 at 5:05 PM Wesley Dillingham 
wrote:

> It will take a couple weeks to a couple months to complete is my best
> guess on 10TB spinners at ~40% full. The cluster should be usable
> throughout the process.
>
> Keep in mind, you should disable the pg autoscaler on any pool which you
> are manually adjusting the pg_num for. Increasing the pg_num is called "pg
> splitting" you can google around for this to see how it will work etc.
>
> There are a few knobs to increase or decrease the aggressiveness of the pg
> split, primarily these are osd_max_backfills and
> target_max_misplaced_ratio.
>
> You can monitor the progress of the split by looking at "ceph osd pool ls
> detail" for the pool you are splitting, for this pool pgp_num will slowly
> increase up until it reaches the pg_num / pg_num_target.
>
> IMO this blog post best covers the issue which you are looking to
> undertake:
> https://ceph.io/en/news/blog/2019/new-in-nautilus-pg-merging-and-autotuning/
>
> Respectfully,
>
> *Wes Dillingham*
> w...@wesdillingham.com
> LinkedIn 
>
>
> On Tue, Jan 30, 2024 at 9:38 AM Michel Niyoyita  wrote:
>
>> Thanks for your advices Wes, below is what ceph osd df tree shows , the
>> increase of pg_num of the production cluster will not affect the
>> performance or crush ? how long it can takes to finish?
>>
>> ceph osd df tree
>> ID  CLASS  WEIGHT REWEIGHT  SIZE RAW USE  DATA  OMAP
>>  META AVAIL%USE   VAR   PGS  STATUS  TYPE NAME
>> -1 433.11841 -  433 TiB  151 TiB67 TiB  364 MiB  210
>> GiB  282 TiB  34.86  1.00-  root default
>> -3 144.37280 -  144 TiB   50 TiB22 TiB  121 MiB   70
>> GiB   94 TiB  34.86  1.00-  host ceph-osd1
>>  2hdd9.02330   1.0  9.0 TiB  2.7 TiB  1021 GiB  5.4 MiB  3.7
>> GiB  6.3 TiB  30.40  0.87   19  up  osd.2
>>  3hdd9.02330   1.0  9.0 TiB  2.7 TiB   931 GiB  4.1 MiB  3.5
>> GiB  6.4 TiB  29.43  0.84   29  up  osd.3
>>  6hdd9.02330   1.0  9.0 TiB  3.3 TiB   1.5 TiB  8.1 MiB  4.5
>> GiB  5.8 TiB  36.09  1.04   20  up  osd.6
>>  9hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.0 TiB  6.6 MiB  3.8
>> GiB  6.2 TiB  30.97  0.89   23  up  osd.9
>> 12hdd9.02330   1.0  9.0 TiB  4.0 TiB   2.3 TiB   13 MiB  6.1
>> GiB  5.0 TiB  44.68  1.28   30  up  osd.12
>> 15hdd9.02330   1.0  9.0 TiB  3.5 TiB   1.8 TiB  9.2 MiB  5.2
>> GiB  5.5 TiB  38.99  1.12   30  up  osd.15
>> 18hdd9.02330   1.0  9.0 TiB  3.0 TiB   1.2 TiB  6.5 MiB  4.0
>> GiB  6.1 TiB  32.80  0.94   21  up  osd.18
>> 22hdd9.02330   1.0  9.0 TiB  3.6 TiB   1.9 TiB   10 MiB  5.4
>> GiB  5.4 TiB  40.25  1.15   22  up  osd.22
>> 25hdd9.02330   1.0  9.0 TiB  3.9 TiB   2.1 TiB   12 MiB  5.7
>> GiB

[ceph-users] Re: 6 pgs not deep-scrubbed in time

2024-01-30 Thread Wesley Dillingham 
You may want to consider upgrading to 16.2.14 before you do the pg split.

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Tue, Jan 30, 2024 at 10:18 AM Michel Niyoyita  wrote:

> I tried that on one of my pool (pool id 3) but the number of pgs not
> deep-scrubbed in time increased also from 55 to 100 but the number of PGs
> was increased. I set also autoscale to off mode. before continue to other
> pools would like to ask if so far there is no negative impact.
>
> ceph -s
>   cluster:
> id: cb0caedc-eb5b-42d1-a34f-96facfda8c27
> health: HEALTH_WARN
> 100 pgs not deep-scrubbed in time
>
>   services:
> mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M)
> mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1
> osd: 48 osds: 48 up (since 11M), 48 in (since 12M)
> rgw: 6 daemons active (6 hosts, 1 zones)
>
>   data:
> pools:   10 pools, 609 pgs
> objects: 6.03M objects, 23 TiB
> usage:   151 TiB used, 282 TiB / 433 TiB avail
> pgs: 603 active+clean
>  4   active+clean+scrubbing+deep
>  2   active+clean+scrubbing
>
>   io:
> client:   96 MiB/s rd, 573 MiB/s wr, 576 op/s rd, 648 op/s wr
>
> root@ceph-osd3:/var/log# ceph df
> --- RAW STORAGE ---
> CLASS SIZEAVAIL USED  RAW USED  %RAW USED
> hdd433 TiB  282 TiB  151 TiB   151 TiB  34.93
> TOTAL  433 TiB  282 TiB  151 TiB   151 TiB  34.93
>
> --- POOLS ---
> POOL   ID  PGS   STORED  OBJECTS USED  %USED  MAX AVAIL
> device_health_metrics   11  1.1 MiB3  3.2 MiB  0 72 TiB
> .rgw.root   2   32  3.7 KiB8   96 KiB  0 72 TiB
> default.rgw.log 3  256  3.6 KiB  204  408 KiB  0 72 TiB
> default.rgw.control 4   32  0 B8  0 B  0 72 TiB
> default.rgw.meta5   32382 B2   24 KiB  0 72 TiB
> volumes 6  128   21 TiB5.74M   62 TiB  22.30 72 TiB
> images  7   32  878 GiB  112.50k  2.6 TiB   1.17 72 TiB
> backups 8   32  0 B0  0 B  0 72 TiB
> vms 9   32  870 GiB  170.73k  2.5 TiB   1.13 72 TiB
> testbench  10   32  0 B0  0 B  0 72 TiB
>
> On Tue, Jan 30, 2024 at 5:05 PM Wesley Dillingham 
> wrote:
>
>> It will take a couple weeks to a couple months to complete is my best
>> guess on 10TB spinners at ~40% full. The cluster should be usable
>> throughout the process.
>>
>> Keep in mind, you should disable the pg autoscaler on any pool which you
>> are manually adjusting the pg_num for. Increasing the pg_num is called "pg
>> splitting" you can google around for this to see how it will work etc.
>>
>> There are a few knobs to increase or decrease the aggressiveness of the
>> pg split, primarily these are osd_max_backfills and
>> target_max_misplaced_ratio.
>>
>> You can monitor the progress of the split by looking at "ceph osd pool ls
>> detail" for the pool you are splitting, for this pool pgp_num will slowly
>> increase up until it reaches the pg_num / pg_num_target.
>>
>> IMO this blog post best covers the issue which you are looking to
>> undertake:
>> https://ceph.io/en/news/blog/2019/new-in-nautilus-pg-merging-and-autotuning/
>>
>> Respectfully,
>>
>> *Wes Dillingham*
>> w...@wesdillingham.com
>> LinkedIn 
>>
>>
>> On Tue, Jan 30, 2024 at 9:38 AM Michel Niyoyita 
>> wrote:
>>
>>> Thanks for your advices Wes, below is what ceph osd df tree shows , the
>>> increase of pg_num of the production cluster will not affect the
>>> performance or crush ? how long it can takes to finish?
>>>
>>> ceph osd df tree
>>> ID  CLASS  WEIGHT REWEIGHT  SIZE RAW USE  DATA  OMAP
>>>  META AVAIL%USE   VAR   PGS  STATUS  TYPE NAME
>>> -1 433.11841 -  433 TiB  151 TiB67 TiB  364 MiB  210
>>> GiB  282 TiB  34.86  1.00-  root default
>>> -3 144.37280 -  144 TiB   50 TiB22 TiB  121 MiB   70
>>> GiB   94 TiB  34.86  1.00-  host ceph-osd1
>>>  2hdd9.02330   1.0  9.0 TiB  2.7 TiB  1021 GiB  5.4 MiB  3.7
>>> GiB  6.3 TiB  30.40  0.87   19  up  osd.2
>>>  3hdd9.02330   1.0  9.0 TiB  2.7 TiB   931 GiB  4.1 MiB  3.5
>>> GiB  6.4 TiB  29.43  0.84   29  up  osd.3
>>>  6hdd9.02330   1.0  9.0 TiB  3.3 TiB   1.5 TiB  8.1 MiB  4.5
>>> GiB  5.8 TiB  36.09  1.04   20  up  osd.6
>>>  9hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.0 TiB  6.6 MiB  3.8
>>> GiB  6.2 TiB  30.97  0.89   23  up  osd.9
>>> 12hdd9.02330   1.0  9.0 TiB  4.0 TiB   2.3 TiB   13 MiB  6.1
>>> GiB  5.0 TiB  44.68  1.28   30  up  osd.12
>>> 15hdd9.02330   1.0  9.0 TiB  3.5 TiB   1.8 TiB  9.2 MiB  5.2
>>> GiB  5.5 TiB

[ceph-users] Re: 6 pgs not deep-scrubbed in time

2024-01-30 Thread Wesley Dillingham 
actually it seems the issue I had in mind was fixed in 16.2.11 so you
should be fine.

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Tue, Jan 30, 2024 at 10:34 AM Wesley Dillingham 
wrote:

> You may want to consider upgrading to 16.2.14 before you do the pg split.
>
> Respectfully,
>
> *Wes Dillingham*
> w...@wesdillingham.com
> LinkedIn 
>
>
> On Tue, Jan 30, 2024 at 10:18 AM Michel Niyoyita 
> wrote:
>
>> I tried that on one of my pool (pool id 3) but the number of pgs not
>> deep-scrubbed in time increased also from 55 to 100 but the number of PGs
>> was increased. I set also autoscale to off mode. before continue to other
>> pools would like to ask if so far there is no negative impact.
>>
>> ceph -s
>>   cluster:
>> id: cb0caedc-eb5b-42d1-a34f-96facfda8c27
>> health: HEALTH_WARN
>> 100 pgs not deep-scrubbed in time
>>
>>   services:
>> mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M)
>> mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1
>> osd: 48 osds: 48 up (since 11M), 48 in (since 12M)
>> rgw: 6 daemons active (6 hosts, 1 zones)
>>
>>   data:
>> pools:   10 pools, 609 pgs
>> objects: 6.03M objects, 23 TiB
>> usage:   151 TiB used, 282 TiB / 433 TiB avail
>> pgs: 603 active+clean
>>  4   active+clean+scrubbing+deep
>>  2   active+clean+scrubbing
>>
>>   io:
>> client:   96 MiB/s rd, 573 MiB/s wr, 576 op/s rd, 648 op/s wr
>>
>> root@ceph-osd3:/var/log# ceph df
>> --- RAW STORAGE ---
>> CLASS SIZEAVAIL USED  RAW USED  %RAW USED
>> hdd433 TiB  282 TiB  151 TiB   151 TiB  34.93
>> TOTAL  433 TiB  282 TiB  151 TiB   151 TiB  34.93
>>
>> --- POOLS ---
>> POOL   ID  PGS   STORED  OBJECTS USED  %USED  MAX
>> AVAIL
>> device_health_metrics   11  1.1 MiB3  3.2 MiB  0 72
>> TiB
>> .rgw.root   2   32  3.7 KiB8   96 KiB  0 72
>> TiB
>> default.rgw.log 3  256  3.6 KiB  204  408 KiB  0 72
>> TiB
>> default.rgw.control 4   32  0 B8  0 B  0 72
>> TiB
>> default.rgw.meta5   32382 B2   24 KiB  0 72
>> TiB
>> volumes 6  128   21 TiB5.74M   62 TiB  22.30 72
>> TiB
>> images  7   32  878 GiB  112.50k  2.6 TiB   1.17 72
>> TiB
>> backups 8   32  0 B0  0 B  0 72
>> TiB
>> vms 9   32  870 GiB  170.73k  2.5 TiB   1.13 72
>> TiB
>> testbench  10   32  0 B0  0 B  0 72
>> TiB
>>
>> On Tue, Jan 30, 2024 at 5:05 PM Wesley Dillingham 
>> wrote:
>>
>>> It will take a couple weeks to a couple months to complete is my best
>>> guess on 10TB spinners at ~40% full. The cluster should be usable
>>> throughout the process.
>>>
>>> Keep in mind, you should disable the pg autoscaler on any pool which you
>>> are manually adjusting the pg_num for. Increasing the pg_num is called "pg
>>> splitting" you can google around for this to see how it will work etc.
>>>
>>> There are a few knobs to increase or decrease the aggressiveness of the
>>> pg split, primarily these are osd_max_backfills and
>>> target_max_misplaced_ratio.
>>>
>>> You can monitor the progress of the split by looking at "ceph osd pool
>>> ls detail" for the pool you are splitting, for this pool pgp_num will
>>> slowly increase up until it reaches the pg_num / pg_num_target.
>>>
>>> IMO this blog post best covers the issue which you are looking to
>>> undertake:
>>> https://ceph.io/en/news/blog/2019/new-in-nautilus-pg-merging-and-autotuning/
>>>
>>> Respectfully,
>>>
>>> *Wes Dillingham*
>>> w...@wesdillingham.com
>>> LinkedIn 
>>>
>>>
>>> On Tue, Jan 30, 2024 at 9:38 AM Michel Niyoyita 
>>> wrote:
>>>
 Thanks for your advices Wes, below is what ceph osd df tree shows , the
 increase of pg_num of the production cluster will not affect the
 performance or crush ? how long it can takes to finish?

 ceph osd df tree
 ID  CLASS  WEIGHT REWEIGHT  SIZE RAW USE  DATA  OMAP
  META AVAIL%USE   VAR   PGS  STATUS  TYPE NAME
 -1 433.11841 -  433 TiB  151 TiB67 TiB  364 MiB
 210 GiB  282 TiB  34.86  1.00-  root default
 -3 144.37280 -  144 TiB   50 TiB22 TiB  121 MiB
  70 GiB   94 TiB  34.86  1.00-  host ceph-osd1
  2hdd9.02330   1.0  9.0 TiB  2.7 TiB  1021 GiB  5.4 MiB
 3.7 GiB  6.3 TiB  30.40  0.87   19  up  osd.2
  3hdd9.02330   1.0  9.0 TiB  2.7 TiB   931 GiB  4.1 MiB
 3.5 GiB  6.4 TiB  29.43  0.84   29  up  osd.3
  6hdd9.02330   1.0  9.0 TiB  3.3 TiB   1.5 TiB  8.1 MiB
 4.5 GiB  5.8 TiB  36.09

[ceph-users] Ceph stretch mode connect to local datacenter

2024-01-30 Thread Oleksandr 34 
Hello.
I have a ceph cluster which works in stretch mode:
*DC1:*
node1 (osd, mon, mgr)
node2 (osd, mon)
node3 (osd, mds)

*DC2:*
node1 (osd, mon, mgr)
node2 (osd, mon)
node3 (osd, mds)

*DC3:*
node1 (mon)

Datacenters are distributions  between different  locations.
I use RBD on my clients.
How can I set up my clients to connect to the local datacenter?
I don't need big traffic between datacenters only for replications, so my
client in DC1 should connect to ceph in DC1. But when something happened
with DC1 my clients in DC1 should still be working. In configs on my
clients I set up all cluster monitors.
Is this possible at all?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

2024-01-30 Thread Wesley Dillingham 
I now concur you should increase the pg_num as a first step for this
cluster. Disable the pg autoscaler for and increase the volumes pool to
pg_num 256. Then likely re-asses and make the next power of 2 jump to 512
and probably beyond.

Keep in mind this is not going to fix your short term deep-scrub issue in
fact it will increase the number of not scrubbed in time PGs until the
pg_num change is complete.  This is because OSDs dont scrub when they are
backfilling.

I would sit on 256 for a couple weeks and let scrubs happen then continue
past 256.

with the ultimate target of around 100-200 PGs per OSD which "ceph osd df
tree" will show you in the PGs column.

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Tue, Jan 30, 2024 at 3:16 AM Michel Niyoyita  wrote:

> Dear team,
>
> below is the output of ceph df command and the ceph version I am running
>
>  ceph df
> --- RAW STORAGE ---
> CLASS SIZEAVAIL USED  RAW USED  %RAW USED
> hdd433 TiB  282 TiB  151 TiB   151 TiB  34.82
> TOTAL  433 TiB  282 TiB  151 TiB   151 TiB  34.82
>
> --- POOLS ---
> POOL   ID  PGS   STORED  OBJECTS USED  %USED  MAX AVAIL
> device_health_metrics   11  1.1 MiB3  3.2 MiB  0 73 TiB
> .rgw.root   2   32  3.7 KiB8   96 KiB  0 73 TiB
> default.rgw.log 3   32  3.6 KiB  209  408 KiB  0 73 TiB
> default.rgw.control 4   32  0 B8  0 B  0 73 TiB
> default.rgw.meta5   32382 B2   24 KiB  0 73 TiB
> volumes 6  128   21 TiB5.68M   62 TiB  22.09 73 TiB
> images  7   32  878 GiB  112.50k  2.6 TiB   1.17 73 TiB
> backups 8   32  0 B0  0 B  0 73 TiB
> vms 9   32  881 GiB  174.30k  2.5 TiB   1.13 73 TiB
> testbench  10   32  0 B0  0 B  0 73 TiB
> root@ceph-mon1:~# ceph --version
> ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific
> (stable)
> root@ceph-mon1:~#
>
> please advise accordingly
>
> Michel
>
> On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder  wrote:
>
> > You will have to look at the output of "ceph df" and make a decision to
> > balance "objects per PG" and "GB per PG". Increase he PG count for the
> > pools with the worst of these two numbers most such that it balances out
> as
> > much as possible. If you have pools that see significantly more user-IO
> > than others, prioritise these.
> >
> > You will have to find out for your specific cluster, we can only give
> > general guidelines. Make changes, run benchmarks, re-evaluate. Take the
> > time for it. The better you know your cluster and your users, the better
> > the end result will be.
> >
> > Best regards,
> > =
> > Frank Schilder
> > AIT Risø Campus
> > Bygning 109, rum S14
> >
> > 
> > From: Michel Niyoyita 
> > Sent: Monday, January 29, 2024 2:04 PM
> > To: Janne Johansson
> > Cc: Frank Schilder; E Taka; ceph-users
> > Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time
> >
> > This is how it is set , if you suggest to make some changes please
> advises.
> >
> > Thank you.
> >
> >
> > ceph osd pool ls detail
> > pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0
> > object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change
> 1407
> > flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application
> > mgr_devicehealth
> > pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 flags
> > hashpspool stripe_width 0 application rgw
> > pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0
> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
> > 1394 flags hashpspool stripe_width 0 application rgw
> > pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0
> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
> > 1395 flags hashpspool stripe_width 0 application rgw
> > pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0
> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
> > 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw
> > pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 108802 lfor
> > 0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 application
> rbd
> > removed_snaps_queue
> >
> [22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c]
> > pool 7 'images' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 32 pgp_num 32

[ceph-users] Re: 6 pgs not deep-scrubbed in time

2024-01-30 Thread Wesley Dillingham 
It will take a couple weeks to a couple months to complete is my best guess
on 10TB spinners at ~40% full. The cluster should be usable throughout the
process.

Keep in mind, you should disable the pg autoscaler on any pool which you
are manually adjusting the pg_num for. Increasing the pg_num is called "pg
splitting" you can google around for this to see how it will work etc.

There are a few knobs to increase or decrease the aggressiveness of the pg
split, primarily these are osd_max_backfills and
target_max_misplaced_ratio.

You can monitor the progress of the split by looking at "ceph osd pool ls
detail" for the pool you are splitting, for this pool pgp_num will slowly
increase up until it reaches the pg_num / pg_num_target.

IMO this blog post best covers the issue which you are looking to
undertake:
https://ceph.io/en/news/blog/2019/new-in-nautilus-pg-merging-and-autotuning/

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Tue, Jan 30, 2024 at 9:38 AM Michel Niyoyita  wrote:

> Thanks for your advices Wes, below is what ceph osd df tree shows , the
> increase of pg_num of the production cluster will not affect the
> performance or crush ? how long it can takes to finish?
>
> ceph osd df tree
> ID  CLASS  WEIGHT REWEIGHT  SIZE RAW USE  DATA  OMAP META
>AVAIL%USE   VAR   PGS  STATUS  TYPE NAME
> -1 433.11841 -  433 TiB  151 TiB67 TiB  364 MiB  210
> GiB  282 TiB  34.86  1.00-  root default
> -3 144.37280 -  144 TiB   50 TiB22 TiB  121 MiB   70
> GiB   94 TiB  34.86  1.00-  host ceph-osd1
>  2hdd9.02330   1.0  9.0 TiB  2.7 TiB  1021 GiB  5.4 MiB  3.7
> GiB  6.3 TiB  30.40  0.87   19  up  osd.2
>  3hdd9.02330   1.0  9.0 TiB  2.7 TiB   931 GiB  4.1 MiB  3.5
> GiB  6.4 TiB  29.43  0.84   29  up  osd.3
>  6hdd9.02330   1.0  9.0 TiB  3.3 TiB   1.5 TiB  8.1 MiB  4.5
> GiB  5.8 TiB  36.09  1.04   20  up  osd.6
>  9hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.0 TiB  6.6 MiB  3.8
> GiB  6.2 TiB  30.97  0.89   23  up  osd.9
> 12hdd9.02330   1.0  9.0 TiB  4.0 TiB   2.3 TiB   13 MiB  6.1
> GiB  5.0 TiB  44.68  1.28   30  up  osd.12
> 15hdd9.02330   1.0  9.0 TiB  3.5 TiB   1.8 TiB  9.2 MiB  5.2
> GiB  5.5 TiB  38.99  1.12   30  up  osd.15
> 18hdd9.02330   1.0  9.0 TiB  3.0 TiB   1.2 TiB  6.5 MiB  4.0
> GiB  6.1 TiB  32.80  0.94   21  up  osd.18
> 22hdd9.02330   1.0  9.0 TiB  3.6 TiB   1.9 TiB   10 MiB  5.4
> GiB  5.4 TiB  40.25  1.15   22  up  osd.22
> 25hdd9.02330   1.0  9.0 TiB  3.9 TiB   2.1 TiB   12 MiB  5.7
> GiB  5.1 TiB  42.94  1.23   22  up  osd.25
> 28hdd9.02330   1.0  9.0 TiB  3.1 TiB   1.4 TiB  7.5 MiB  4.1
> GiB  5.9 TiB  34.87  1.00   21  up  osd.28
> 32hdd9.02330   1.0  9.0 TiB  2.7 TiB  1017 GiB  4.8 MiB  3.7
> GiB  6.3 TiB  30.36  0.87   27  up  osd.32
> 35hdd9.02330   1.0  9.0 TiB  3.0 TiB   1.3 TiB  7.2 MiB  4.2
> GiB  6.0 TiB  33.73  0.97   21  up  osd.35
> 38hdd9.02330   1.0  9.0 TiB  3.1 TiB   1.4 TiB  7.3 MiB  4.1
> GiB  5.9 TiB  34.57  0.99   24  up  osd.38
> 41hdd9.02330   1.0  9.0 TiB  2.9 TiB   1.2 TiB  6.2 MiB  4.0
> GiB  6.1 TiB  32.49  0.93   24  up  osd.41
> 44hdd9.02330   1.0  9.0 TiB  3.1 TiB   1.4 TiB  7.3 MiB  4.4
> GiB  5.9 TiB  34.87  1.00   29  up  osd.44
> 47hdd9.02330   1.0  9.0 TiB  2.7 TiB  1016 GiB  5.4 MiB  3.6
> GiB  6.3 TiB  30.35  0.87   23  up  osd.47
> -7 144.37280 -  144 TiB   50 TiB22 TiB  122 MiB   70
> GiB   94 TiB  34.86  1.00-  host ceph-osd2
>  1hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.1 TiB  5.7 MiB  3.8
> GiB  6.2 TiB  31.00  0.89   27  up  osd.1
>  5hdd9.02330   1.0  9.0 TiB  3.2 TiB   1.5 TiB  7.3 MiB  4.5
> GiB  5.8 TiB  35.45  1.02   27  up  osd.5
>  8hdd9.02330   1.0  9.0 TiB  3.3 TiB   1.6 TiB  8.3 MiB  4.7
> GiB  5.7 TiB  36.85  1.06   30  up  osd.8
> 10hdd9.02330   1.0  9.0 TiB  3.1 TiB   1.4 TiB  7.5 MiB  4.5
> GiB  5.9 TiB  34.87  1.00   20  up  osd.10
> 13hdd9.02330   1.0  9.0 TiB  3.6 TiB   1.8 TiB   10 MiB  5.3
> GiB  5.4 TiB  39.63  1.14   27  up  osd.13
> 16hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.1 TiB  6.0 MiB  3.8
> GiB  6.2 TiB  31.01  0.89   19  up  osd.16
> 19hdd9.02330   1.0  9.0 TiB  3.0 TiB   1.2 TiB  6.4 MiB  4.0
> GiB  6.1 TiB  32.77  0.94   21  up  osd.19
> 21hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.1 TiB  5.5 MiB  3.7
> GiB  6.2 TiB  31.58  0.91   26  up  osd.21
> 24hdd9.02330

[ceph-users] Re: 6 pgs not deep-scrubbed in time

2024-01-30 Thread Michel Niyoyita 
Thanks for your advices Wes, below is what ceph osd df tree shows , the
increase of pg_num of the production cluster will not affect the
performance or crush ? how long it can takes to finish?

ceph osd df tree
ID  CLASS  WEIGHT REWEIGHT  SIZE RAW USE  DATA  OMAP META
   AVAIL%USE   VAR   PGS  STATUS  TYPE NAME
-1 433.11841 -  433 TiB  151 TiB67 TiB  364 MiB  210
GiB  282 TiB  34.86  1.00-  root default
-3 144.37280 -  144 TiB   50 TiB22 TiB  121 MiB   70
GiB   94 TiB  34.86  1.00-  host ceph-osd1
 2hdd9.02330   1.0  9.0 TiB  2.7 TiB  1021 GiB  5.4 MiB  3.7
GiB  6.3 TiB  30.40  0.87   19  up  osd.2
 3hdd9.02330   1.0  9.0 TiB  2.7 TiB   931 GiB  4.1 MiB  3.5
GiB  6.4 TiB  29.43  0.84   29  up  osd.3
 6hdd9.02330   1.0  9.0 TiB  3.3 TiB   1.5 TiB  8.1 MiB  4.5
GiB  5.8 TiB  36.09  1.04   20  up  osd.6
 9hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.0 TiB  6.6 MiB  3.8
GiB  6.2 TiB  30.97  0.89   23  up  osd.9
12hdd9.02330   1.0  9.0 TiB  4.0 TiB   2.3 TiB   13 MiB  6.1
GiB  5.0 TiB  44.68  1.28   30  up  osd.12
15hdd9.02330   1.0  9.0 TiB  3.5 TiB   1.8 TiB  9.2 MiB  5.2
GiB  5.5 TiB  38.99  1.12   30  up  osd.15
18hdd9.02330   1.0  9.0 TiB  3.0 TiB   1.2 TiB  6.5 MiB  4.0
GiB  6.1 TiB  32.80  0.94   21  up  osd.18
22hdd9.02330   1.0  9.0 TiB  3.6 TiB   1.9 TiB   10 MiB  5.4
GiB  5.4 TiB  40.25  1.15   22  up  osd.22
25hdd9.02330   1.0  9.0 TiB  3.9 TiB   2.1 TiB   12 MiB  5.7
GiB  5.1 TiB  42.94  1.23   22  up  osd.25
28hdd9.02330   1.0  9.0 TiB  3.1 TiB   1.4 TiB  7.5 MiB  4.1
GiB  5.9 TiB  34.87  1.00   21  up  osd.28
32hdd9.02330   1.0  9.0 TiB  2.7 TiB  1017 GiB  4.8 MiB  3.7
GiB  6.3 TiB  30.36  0.87   27  up  osd.32
35hdd9.02330   1.0  9.0 TiB  3.0 TiB   1.3 TiB  7.2 MiB  4.2
GiB  6.0 TiB  33.73  0.97   21  up  osd.35
38hdd9.02330   1.0  9.0 TiB  3.1 TiB   1.4 TiB  7.3 MiB  4.1
GiB  5.9 TiB  34.57  0.99   24  up  osd.38
41hdd9.02330   1.0  9.0 TiB  2.9 TiB   1.2 TiB  6.2 MiB  4.0
GiB  6.1 TiB  32.49  0.93   24  up  osd.41
44hdd9.02330   1.0  9.0 TiB  3.1 TiB   1.4 TiB  7.3 MiB  4.4
GiB  5.9 TiB  34.87  1.00   29  up  osd.44
47hdd9.02330   1.0  9.0 TiB  2.7 TiB  1016 GiB  5.4 MiB  3.6
GiB  6.3 TiB  30.35  0.87   23  up  osd.47
-7 144.37280 -  144 TiB   50 TiB22 TiB  122 MiB   70
GiB   94 TiB  34.86  1.00-  host ceph-osd2
 1hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.1 TiB  5.7 MiB  3.8
GiB  6.2 TiB  31.00  0.89   27  up  osd.1
 5hdd9.02330   1.0  9.0 TiB  3.2 TiB   1.5 TiB  7.3 MiB  4.5
GiB  5.8 TiB  35.45  1.02   27  up  osd.5
 8hdd9.02330   1.0  9.0 TiB  3.3 TiB   1.6 TiB  8.3 MiB  4.7
GiB  5.7 TiB  36.85  1.06   30  up  osd.8
10hdd9.02330   1.0  9.0 TiB  3.1 TiB   1.4 TiB  7.5 MiB  4.5
GiB  5.9 TiB  34.87  1.00   20  up  osd.10
13hdd9.02330   1.0  9.0 TiB  3.6 TiB   1.8 TiB   10 MiB  5.3
GiB  5.4 TiB  39.63  1.14   27  up  osd.13
16hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.1 TiB  6.0 MiB  3.8
GiB  6.2 TiB  31.01  0.89   19  up  osd.16
19hdd9.02330   1.0  9.0 TiB  3.0 TiB   1.2 TiB  6.4 MiB  4.0
GiB  6.1 TiB  32.77  0.94   21  up  osd.19
21hdd9.02330   1.0  9.0 TiB  2.8 TiB   1.1 TiB  5.5 MiB  3.7
GiB  6.2 TiB  31.58  0.91   26  up  osd.21
24hdd9.02330   1.0  9.0 TiB  2.6 TiB   855 GiB  4.7 MiB  3.3
GiB  6.4 TiB  28.61  0.82   19  up  osd.24
27hdd9.02330   1.0  9.0 TiB  3.7 TiB   1.9 TiB   10 MiB  5.2
GiB  5.3 TiB  40.84  1.17   24  up  osd.27
30hdd9.02330   1.0  9.0 TiB  3.2 TiB   1.4 TiB  7.5 MiB  4.5
GiB  5.9 TiB  35.16  1.01   22  up  osd.30
33hdd9.02330   1.0  9.0 TiB  3.1 TiB   1.4 TiB  8.6 MiB  4.3
GiB  5.9 TiB  34.59  0.99   23  up  osd.33
36hdd9.02330   1.0  9.0 TiB  3.4 TiB   1.7 TiB   10 MiB  5.0
GiB  5.6 TiB  38.17  1.09   25  up  osd.36
39hdd9.02330   1.0  9.0 TiB  3.4 TiB   1.7 TiB  8.5 MiB  5.1
GiB  5.6 TiB  37.79  1.08   31  up  osd.39
42hdd9.02330   1.0  9.0 TiB  3.6 TiB   1.8 TiB   10 MiB  5.2
GiB  5.4 TiB  39.68  1.14   23  up  osd.42
45hdd9.02330   1.0  9.0 TiB  2.7 TiB   964 GiB  5.1 MiB  3.5
GiB  6.3 TiB  29.78  0.85   21  up  osd.45
-5 144.37280 -  144 TiB   50 TiB22 TiB  121 MiB   70
GiB   94 TiB  34.86  1.00-  host ceph-osd3
 0hdd9.02330   1.0  9.0 TiB  2.7 TiB   934

[ceph-users] Re: pacific 16.2.15 QE validation status

2024-01-30 Thread Nizamudeen A 
dashboard looks good! approved.

Regards,
Nizam

On Tue, Jan 30, 2024 at 3:09 AM Yuri Weinstein  wrote:

> Details of this release are summarized here:
>
> https://tracker.ceph.com/issues/64151#note-1
>
> Seeking approvals/reviews for:
>
> rados - Radek, Laura, Travis, Ernesto, Adam King
> rgw - Casey
> fs - Venky
> rbd - Ilya
> krbd - in progress
>
> upgrade/nautilus-x (pacific) - Casey PTL (regweed tests failed)
> upgrade/octopus-x (pacific) - Casey PTL (regweed tests failed)
>
> upgrade/pacific-x (quincy) - in progress
> upgrade/pacific-p2p - Ilya PTL (maybe rbd related?)
>
> ceph-volume - Guillaume
>
> TIA
> YuriW
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] cephfs inode backtrace information

2024-01-30 Thread Dietmar Rieder 

Hello,

I have a question regarding the default pool of a cephfs.

According to the docs it is recommended to use a fast ssd replicated 
pool as default pool for cephfs. I'm asking what are the space 
requirements for storing the inode backtrace information?


Let's say I have a 85 TiB replicated ssd pool (hot data) and as 3 PiB EC 
data pool (cold data).


Does it make sense to create a third pool as default pool which only 
holds the inode backtrace information (what would be a good size), or is 
it OK to use the ssd pool as default pool?


Thanks
   Dietmar



OpenPGP_signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: pacific 16.2.15 QE validation status

2024-01-30 Thread Guillaume Abrioux 
Hi Yuri,

The ceph-volume failure is a valid bug.
Investigating for the root cause of it and will submit a patch.

Thanks!

--
Guillaume Abrioux
Software Engineer

From: Yuri Weinstein 
Date: Monday, 29 January 2024 at 22:38
To: dev , ceph-users 
Subject: [EXTERNAL] [ceph-users] pacific 16.2.15 QE validation status
Details of this release are summarized here:

https://tracker.ceph.com/issues/64151#note-1

Seeking approvals/reviews for:

rados - Radek, Laura, Travis, Ernesto, Adam King
rgw - Casey
fs - Venky
rbd - Ilya
krbd - in progress

upgrade/nautilus-x (pacific) - Casey PTL (regweed tests failed)
upgrade/octopus-x (pacific) - Casey PTL (regweed tests failed)

upgrade/pacific-x (quincy) - in progress
upgrade/pacific-p2p - Ilya PTL (maybe rbd related?)

ceph-volume - Guillaume

TIA
YuriW
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

Unless otherwise stated above:

Compagnie IBM France
Siège Social : 17, avenue de l'Europe, 92275 Bois-Colombes Cedex
RCS Nanterre 552 118 465
Forme Sociale : S.A.S.
Capital Social : 664 069 390,60 €
SIRET : 552 118 465 03644 - Code NAF 6203Z
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing?

2024-01-30 Thread Jan Marek 
Hello Sridhar,

at Saturday I've finished upgrade proces to 18.2.1.

Cluster is now in HEALTH_OK state and performs well.

According to my colleagues there are lower latences and good
throughput.

On OSD nodes there is relative low I/O activity.

I still have mClock profile "high_client_ops".

When I was stucked in the upgrade process, I had in logs so many
records, see attached file. Since upgrade is complete, this
messages went away... Can be this reason of poor
performance?

Sincerely
Jan Marek

Dne Čt, led 25, 2024 at 02:31:41 CET napsal(a) Jan Marek:
> Hello Sridhar,
> 
> Dne Čt, led 25, 2024 at 09:53:26 CET napsal(a) Sridhar Seshasayee:
> > Hello Jan,
> > 
> > Meaning of my previous post was, that CEPH cluster didn't fulfill
> > my needs and, although I had set mClock profile to
> > "high_client_ops" (because I have a plenty of time to rebalancing
> > and scrubbing), my clients went to problems.
> > 
> > As far as the question around mClock is concerned, there are further
> > improvements in the works to handle QoS between client ops and
> > background scrub ops. This should help address the issue you are
> > currently facing. See PR: https://github.com/ceph/ceph/pull/51171
> > for more information.
> > Also, it would be helpful to know the Ceph version you are currently using.
> 
> thanks for your reply.
> 
> I've just in process upgrade between 17.2.6 and 18.2.1 (you can
> see my previous posts about stuck in upgrade to reef).
> 
> Maybe this was cause of my problem...
> 
> Now I've tried give rest to the cluster to do some "background"
> tasks (and it seems, that this was correct, because on my hosts
> there is around 50-100MBps read and cca 10-50MBps write traffic -
> cca 1/4-1/2 of previous load).
> 
> At Saturday I will change some settings on networking and I will
> try to start upgrade process, maybe with --limit=1, to be "soft"
> for cluster and for our clients...
> 
> > -Sridhar
> 
> Sincerely
> Jan Marek
> -- 
> Ing. Jan Marek
> University of South Bohemia
> Academic Computer Centre
> Phone: +420389032080
> http://www.gnu.org/philosophy/no-word-attachments.cs.html



> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io


-- 
Ing. Jan Marek
University of South Bohemia
Academic Computer Centre
Phone: +420389032080
http://www.gnu.org/philosophy/no-word-attachments.cs.html


signature.asc
Description: PGP signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW crashes when rgw_enable_ops_log is enabled

2024-01-30 Thread Marc Singer 

Hi

The issue is open: https://tracker.ceph.com/issues/64244

If you could take a look or let me know what are the next steps I would 
be super grateful.


In the meantime I will try to increase the read throughput.

Thanks

Marc

On 1/26/24 15:23, Matt Benjamin wrote:

Hi Marc,

1. if you can, yes, create a tracker issue on tracker.ceph.com?
2. you might be able to get more throughput with (some number) of
additional threads;  the first thing I would try is prioritization (nice)

regards,

Matt


On Fri, Jan 26, 2024 at 6:08 AM Marc Singer  wrote:


Hi Matt

Thanks for your answer.

Should I open a bug report then?

How would I be able to read more from it? Have multiple threads access
it and read from it simultaneously?

Marc

On 1/25/24 20:25, Matt Benjamin wrote:

Hi Marc,

No, the only thing you need to do with the Unix socket is to keep
reading from it.  So it probably is getting backlogged.  And while you
could arrange things to make that less likely, you likely can't make
it impossible, so there's a bug here.

Matt

On Thu, Jan 25, 2024 at 10:52 AM Marc Singer

wrote:

 Hi

 I am using a unix socket client to connect with it and read the data
 from it.
 Do I need to do anything like signal the socket that this data has
 been
 read? Or am I not reading fast enough and data is backing up?

 What I am also noticing that at some point (probably after something
 with the ops socket happens), the log level seems to increase for
 some
 reason? I did not find anything in the logs yet why this would be
 the case.

 *Normal:*

 2024-01-25T15:47:58.444+ 7fe98a5c0b00  1 == starting new
 request
 req=0x7fe98712c720 =
 2024-01-25T15:47:58.548+ 7fe98b700b00  1 == req done
 req=0x7fe98712c720 op status=0 http_status=200
 latency=0.104001537s ==
 2024-01-25T15:47:58.548+ 7fe98b700b00  1 beast: 0x7fe98712c720:
 redacted - redacted [25/Jan/2024:15:47:58.444 +] "PUT
 /redacted/redacted/chunks/27/27242/27242514_10_4194304 HTTP/1.1" 200
 4194304 - "redacted" - latency=0.104001537s

 *Close before crashing:
 *

-509> 2024-01-25T14:54:31.588+ 7f5186648b00  1 == starting
 new request req=0x7f517ffca720 =
-508> 2024-01-25T14:54:31.588+ 7f5186648b00  2 req
 2568229052387020224 0.0s initializing for trans_id =
 tx023a42eb7515dcdc0-0065b27627-823feaa-central
-507> 2024-01-25T14:54:31.588+ 7f5186648b00  2 req
 2568229052387020224 0.0s getting op 1
-506> 2024-01-25T14:54:31.588+ 7f5186648b00  2 req
 2568229052387020224 0.0ss3:put_obj  verifying requester
-505> 2024-01-25T14:54:31.588+ 7f5186648b00  2 req
 2568229052387020224 0.0ss3:put_obj  normalizing buckets
 and tenants
-504> 2024-01-25T14:54:31.588+ 7f5186648b00  2 req
 2568229052387020224 0.0ss3:put_obj  init permissions
-503> 2024-01-25T14:54:31.588+ 7f5186648b00  2 req
 2568229052387020224 0.0ss3:put_obj  recalculating target
-502> 2024-01-25T14:54:31.588+ 7f5186648b00  2 req
 2568229052387020224 0.0ss3:put_obj  reading permissions
-501> 2024-01-25T14:54:31.588+ 7f5186648b00  2 req
 2568229052387020224 0.0ss3:put_obj  init op
-500> 2024-01-25T14:54:31.588+ 7f5186648b00  2 req
 2568229052387020224 0.0ss3:put_obj  verifying op mask
-499> 2024-01-25T14:54:31.588+ 7f5186648b00  2 req
 2568229052387020224 0.0ss3:put_obj  verifying op permissions
-498> 2024-01-25T14:54:31.588+ 7f5186648b00  5 req
 2568229052387020224 0.0ss3:put_obj  Searching permissions for
 identity=rgw::auth::SysReqApplier  ->
 rgw::auth::LocalApplier(acct_user=redacted, acct_name=redacted,
 subuser=, perm_mask=15, is_admin=0) mask=50
-497> 2024-01-25T14:54:31.588+ 7f5186648b00  5 req
 2568229052387020224 0.0ss3:put_obj  Searching permissions for
 uid=redacted
-496> 2024-01-25T14:54:31.588+ 7f5186648b00  5 req
 2568229052387020224 0.0ss3:put_obj  Found permission: 15
-495> 2024-01-25T14:54:31.588+ 7f5186648b00  5 req
 2568229052387020224 0.0ss3:put_obj  Searching permissions for
 group=1 mask=50
-494> 2024-01-25T14:54:31.588+ 7f5186648b00  5 req
 2568229052387020224 0.0ss3:put_obj  Permissions for group
 not found
-493> 2024-01-25T14:54:31.588+ 7f5186648b00  5 req
 2568229052387020224 0.0ss3:put_obj  Searching permissions for
 group=2 mask=50
-492> 2024-01-25T14:54:31.588+ 7f5186648b00  5 req
 2568229052387020224 0.0ss3:put_obj  Permissions for group
 not found
-491> 2024-01-25T14:54:31.588+ 7f5186648b00  5 req
 2568229052387020224 0.0ss3:put_obj  -- Getting permissions
 done
 for

[ceph-users] Re: pacific 16.2.15 QE validation status

2024-01-30 Thread Yuri Weinstein 
Update.
Seeking approvals/reviews for:

rados - Radek, Laura, Travis, Ernesto, Adam King
rgw - Casey
fs - Venky
rbd - Ilya
krbd - Ilya

upgrade/nautilus-x (pacific) - Casey PTL (regweed tests failed)
upgrade/octopus-x (pacific) - Casey PTL (regweed tests failed)

upgrade/pacific-x (quincy) - blocked by
https://tracker.ceph.com/issues/64256 (Laura, Dan, Adam pls take a
look)
upgrade/pacific-p2p - Ilya PTL (maybe rbd related?)

ceph-volume - Guillaume is fixing

On Mon, Jan 29, 2024 at 1:38 PM Yuri Weinstein  wrote:
>
> Details of this release are summarized here:
>
> https://tracker.ceph.com/issues/64151#note-1
>
> Seeking approvals/reviews for:
>
> rados - Radek, Laura, Travis, Ernesto, Adam King
> rgw - Casey
> fs - Venky
> rbd - Ilya
> krbd - in progress
>
> upgrade/nautilus-x (pacific) - Casey PTL (regweed tests failed)
> upgrade/octopus-x (pacific) - Casey PTL (regweed tests failed)
>
> upgrade/pacific-x (quincy) - in progress
> upgrade/pacific-p2p - Ilya PTL (maybe rbd related?)
>
> ceph-volume - Guillaume
>
> TIA
> YuriW
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io