[ceph-users] Re: [RGW] Strange issue of multipart object

[Konstantin Shalygin]

Xin chao, 

The Pacific latest (16.2.15) have multiple multipart issue fixes (for example 
[1]), I suggest to upgrade your release for start


k
[1] https://tracker.ceph.com/issues/56673
Sent from my iPhone

> On 18 Jun 2024, at 10:32, Huy Nguyen  wrote:
> 
> Hi,
> I'm using Ceph v16.2.13. Using `radosgw-admin bucket list`, I can see there 
> are 2 multipart objects in a bucket. But I can not show it on boto3 
> (list_multipart_uploads) or s3 browser.
> 
> What I have tried:
> 
> - bucket check: this command can still see the multipart objects but it does 
> nothing, nothing change
> radosgw-admin bucket check --bucket default-standard-bucket --check-objects 
> --fix
> [
>"_multipart_Elearning_Setup_v2.9.zip.QXXXv1NdT7k3xyxqmEeE7x5k-933wzcQ.3",  
>   
> 
>"_multipart_Elearning_Setup_v2.9.zip.dUxF-_13ouu7JYqpx2kTwX1pM6BOK7BN.2"   
>   
> 
> ]
> 
> - rados rm: it returns no such file
> rados -p data.pool rm 
> 17a4ce99-009e-40f2-a2d2-2afc218ebd9b.4817457860.57__shadow_Elearning_Setup_v2.9.zip.QXXXv1NdT7k3xyxqmEeE7x5k-933wzcQ.3_1
> error removing 
> data.pool>17a4ce99-009e-40f2-a2d2-2afc218ebd9b.4817457860.57__shadow_Elearning_Setup_v2.9.zip.QXXXv1NdT7k3xyxqmEeE7x5k-933wzcQ.3_1:
>  (2) No such file or directory
> 
> Does anyone have any idea?
> Thanks
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Unable to Install librados2 18.2.0 on RHEL 7 from Ceph Repository

[Konstantin Shalygin]

Hi,

The last release for EL7 is Octopus (version 15), you try to catch version 18


k
Sent from my iPhone

> On 29 May 2024, at 22:34, abdel.doui...@gmail.com wrote:
> 
> The Ceph repository at https://download.ceph.com/ does not seem to have the 
> librados2 package version 18.2.0 for RHEL 7. The directory  
> https://download.ceph.com/rpm-18.2.0/el7/ is empty, and the specific 
> package URL 
> https://download.ceph.com/rpm-18.2.0/el7/x86_64/librados2-18.2.0-0.el7.x86_64.rpm
>  returns a 404 error.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: unknown PGs after adding hosts in different subtree

[Konstantin Shalygin]

Hi Eugen

> On 21 May 2024, at 15:26, Eugen Block  wrote:
> 
> step set_choose_tries 100

I think you should try to increase set_choose_tries to 200
Last year we had an Pacific EC 8+2 deployment of 10 racks. And even with 50 
hosts, the value of 100 not worked for us


k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph client cluster compatibility

[Konstantin Shalygin]

Hi,

Yes, like it always do


k
Sent from my iPhone

> On 2 May 2024, at 07:09, Nima AbolhassanBeigi 
>  wrote:
> 
> We are trying to upgrade our OS version from ubuntu 18.04 to ubuntu 22.04.
> Our ceph cluster version is 16.2.13 (pacific).
> 
> The problem is that the ubuntu packages for the ceph pacific release will
> not be supported for ubuntu 22.04. We were wondering if the ceph client
> (version 18.2, reef) on ubuntu 22.04 can work with lower version clusters.
> 
> Thanks in advance
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph image delete error - NetHandler create_socket couldnt create socket

[Konstantin Shalygin]

Hi,

> On 19 Apr 2024, at 10:39, Pardhiv Karri  wrote:
> 
> Thank you for the reply. I tried setting ulimit to 32768 when I saw 25726
> number in lsof output and then after 2 disks deletion again it got an error
> and checked lsof and which is above 35000.  I'm not sure how to handle it.
> I rebooted the monitor node, but the open files kept growing.
> 
> root@ceph-mon01 ~# lsof | wc -l
> 49296
> root@ceph-mon01 ~#

This means that is not a Ceph problem. Is a problem in this system at all


k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph image delete error - NetHandler create_socket couldnt create socket

[Konstantin Shalygin]

Hi,

Your shell seems reached the default file discriptors limit (1024 mostly) and 
your cluster maybe more than 1000 OSD

Try to set command `ulimit -n 10240` before rbd rm task


k
Sent from my iPhone

> On 18 Apr 2024, at 23:50, Pardhiv Karri  wrote:
> 
> Hi,
> 
> Trying to delete images in a Ceph pool is causing errors in one of
> the clusters. I rebooted all the monitor nodes sequentially to see if the
> error went away, but it still persists. What is the best way to fix this?
> The Ceph cluster is in an OK state, with no rebalancing or scrubbing
> happening (I did set the noscrub and deep-noscrub flags) and also no load
> on the cluster, very few IO.
> 
> root@ceph-mon01 ~# rbd rm 000dca3d-4f2b-4033-b8f5-95458e0c3444_disk_delete
> -p compute
> Removing image: 31% complete...2024-04-18 20:42:52.525135 7f6de0c79700 -1
> NetHandler create_socket couldn't create socket (24) Too many open files
> Removing image: 32% complete...2024-04-18 20:42:52.539882 7f6de9c7b700 -1
> NetHandler create_socket couldn't create socket (24) Too many open files
> 2024-04-18 20:42:52.541508 7f6de947a700 -1 NetHandler create_socket
> couldn't create socket (24) Too many open files
> 2024-04-18 20:42:52.546613 7f6de0c79700 -1 NetHandler create_socket
> couldn't create socket (24) Too many open files
> 2024-04-18 20:42:52.558133 7f6de9c7b700 -1 NetHandler create_socket
> couldn't create socket (24) Too many open files
> 2024-04-18 20:42:52.573819 7f6de947a700 -1 NetHandler create_socket
> couldn't create socket (24) Too many open files
> 2024-04-18 20:42:52.589733 7f6de0c79700 -1 NetHandler create_socket
> couldn't create socket (24) Too many open files
> Removing image: 33% complete...2024-04-18 20:42:52.643489 7f6de9c7b700 -1
> NetHandler create_socket couldn't create socket (24) Too many open files
> 2024-04-18 20:42:52.727262 7f6de0c79700 -1 NetHandler create_socket
> couldn't create socket (24) Too many open files
> 2024-04-18 20:42:52.737135 7f6de9c7b700 -1 NetHandler create_socket
> couldn't create socket (24) Too many open files
> 2024-04-18 20:42:52.743292 7f6de947a700 -1 NetHandler create_socket
> couldn't create socket (24) Too many open files
> 2024-04-18 20:42:52.746167 7f6de0c79700 -1 NetHandler create_socket
> couldn't create socket (24) Too many open files
> 2024-04-18 20:42:52.757404 7f6de9c7b700 -1 NetHandler create_socket
> couldn't create socket (24) Too many open files
> Removing image: 34% complete...2024-04-18 20:42:52.773182 7f6de947a700 -1
> NetHandler create_socket couldn't create socket (24) Too many open files
> 2024-04-18 20:42:52.773222 7f6de947a700 -1 NetHandler create_socket
> couldn't create socket (24) Too many open files
> 2024-04-18 20:42:52.789847 7f6de0c79700 -1 NetHandler create_socket
> couldn't create socket (24) Too many open files
> 2024-04-18 20:42:52.844201 7f6de9c7b700 -1 NetHandler create_socket
> couldn't create socket (24) Too many open files
> 
> ^C
> root@ceph-mon01 ~#
> 
> 
> Thanks,
> Pardh
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Client kernel crashes on cephfs access

[Konstantin Shalygin]

Hi Xiubo,

Seems patch already landed to kernel 6.8.7, thanks!


k
Sent from my iPhone

> On 18 Apr 2024, at 05:31, Xiubo Li  wrote:
> 
> Hi Konstantin,
> 
> We have fixed it, please see 
> https://patchwork.kernel.org/project/ceph-devel/list/?series=842682=both.
> 
> - Xiubo
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Client kernel crashes on cephfs access

[Konstantin Shalygin]

Hi,

> On 9 Apr 2024, at 04:07, Xiubo Li  wrote:
> 
> Thanks for reporting this, I generated one patch to fix it. Will send it out 
> after testing is done.

Trace from our users, but from mainline kernel. Look like as trace above


kernel: [ cut here ]
kernel: list_add corruption. next->prev should be prev (8fb9c22139b8), but 
was 8fb9c18910c0. (next=8fb9c18910c0).
kernel: WARNING: CPU: 2 PID: 861985 at lib/list_debug.c:29 
__list_add_valid_or_report+0x6a/0xa0
kernel: Modules linked in: cbc ceph libceph libcrc32c dns_resolver netfs 
intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul polyval_clmulni 
polyval_generic gf128mul ghash_clmulni_intel sha512_ssse3 sha256_ssse3 
sha1_ssse3 cfg80211 aesni_intel crypto_simd cryptd rapl rfkill joydev mousedev 
psmouse i2c_piix4 pcspkr mac_hid dm_mod fuse loop nfnetlink qemu_fw_cfg 
ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 virtio_net 
net_failover virtio_rng failover virtio_scsi virtio_balloon virtio_console 
hid_generic usbhid sr_mod cdrom serio_raw ata_generic atkbd pata_acpi 
virtio_pci dr>
kernel: CPU: 2 PID: 861985 Comm: kworker/u36:4 Tainted: G  D W  
6.8.4-arch1-1 #1 7ea0d8fced45b5f098eb034690645970f116c34c
kernel: Hardware name: oVirt RHEL, BIOS 1.14.0-1.el8 04/01/2014
kernel: Workqueue: writeback wb_workfn (flush-ceph-1)
kernel: RIP: 0010:__list_add_valid_or_report+0x6a/0xa0
kernel: Code: a4 ff 0f 0b 31 c0 c3 cc cc cc cc 48 c7 c7 40 95 a6 8e e8 d9 31 a4 
ff 0f 0b eb e9 48 89 c1 48 c7 c7 68 95 a6 8e e8 c6 31 a4 ff <0f> 0b eb d6 48 89 
d1 48 89 c6 4c 89 c2 48 c7 c7 b8 95 a6 8e e8 ad
kernel: RSP: 0018:afb5c4e03b70 EFLAGS: 00010286
kernel: RAX:  RBX: 8fb9f1f7e820 RCX: 0027
kernel: RDX: 8fbaf7aa19c8 RSI: 0001 RDI: 8fbaf7aa19c0
kernel: RBP: 8fb9f1f7ead8 R08:  R09: afb5c4e03a00
kernel: R10: afb5c4e039f8 R11: 0003 R12: 8fb9f1f7eca0
kernel: R13: 8fb9c2213800 R14: 8fb9c22139c8 R15: 0001
kernel: FS:  () GS:8fbaf7a8() 
knlGS:
kernel: CS:  0010 DS:  ES:  CR0: 80050033
kernel: CR2: 7dcc9182fd08 CR3: 00010bc78005 CR4: 003706f0
kernel: DR0:  DR1:  DR2: 
kernel: DR3:  DR6: fffe0ff0 DR7: 0400
kernel: Call Trace:
kernel:  
kernel:  ? __list_add_valid_or_report+0x6a/0xa0
kernel:  ? __warn+0x81/0x130
kernel:  ? __list_add_valid_or_report+0x6a/0xa0
kernel:  ? report_bug+0x171/0x1a0
kernel:  ? console_unlock+0x78/0x120
kernel:  ? handle_bug+0x3c/0x80
kernel:  ? exc_invalid_op+0x17/0x70
kernel:  ? asm_exc_invalid_op+0x1a/0x20
kernel:  ? __list_add_valid_or_report+0x6a/0xa0
kernel:  ceph_write_inode+0x35b/0x390 [ceph 
e22e5e8fbb2e717a1d900c81c3b686cc119b9d81]
kernel:  ? do_writepages+0x1bb/0x1e0
kernel:  __writeback_single_inode+0x2a9/0x360
kernel:  writeback_sb_inodes+0x1ed/0x4b0
kernel:  __writeback_inodes_wb+0x4c/0xf0
kernel:  wb_writeback+0x298/0x310
kernel:  wb_workfn+0x368/0x510
kernel:  process_one_work+0x178/0x350
kernel:  worker_thread+0x30f/0x450
kernel:  ? __pfx_worker_thread+0x10/0x10
kernel:  kthread+0xe5/0x120
kernel:  ? __pfx_kthread+0x10/0x10
kernel:  ret_from_fork+0x31/0x50
kernel:  ? __pfx_kthread+0x10/0x10
kernel:  ret_from_fork_asm+0x1b/0x30
kernel:  
kernel: ---[ end trace  ]---



k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Impact of large PG splits

[Konstantin Shalygin]

> On 10 Apr 2024, at 01:00, Eugen Block  wrote:
> 
> I appreciate your message, it really sounds tough (9 months, really?!). But 
> thanks for the reassurance :-)

Yes, the total "make this project great again" tooks 16 month, I think. This my 
work

First problem after 1M objects in PG was a deletion [1]. It's just impossible 
to delete objects for the 'stray' PG
The second was - the code, that cares about nearfull & backfillfull just don't 
work for this OSD [2], because code use DATA field (the objects), instead RAW 
field (the DATA + RocksDB database) for computations
The third was minor, but WTF statistics metric issue [3]
And the last but not least (and still present in master) - when lock object 
acquired, this crashes replica OSD's in acting set, when object is absent on 
primary OSD [4]. This may ruin client IO until OSD's restart & recovery

For current time, not all collection_list fixes was merged [5], but since 
14.2.22 much better than before...

> They don’t have any other options so we’ll have to start that process anyway, 
> probably tomorrow. We’ll see how it goes…


Yes, you just have to start, and then we’ll see


Thanks,
k

[1] https://tracker.ceph.com/issues/47044 + 
https://tracker.ceph.com/issues/45765 -> https://tracker.ceph.com/issues/50466
[2] https://tracker.ceph.com/issues/50533
[3] https://tracker.ceph.com/issues/52512
[4] https://tracker.ceph.com/issues/52513
[5] https://tracker.ceph.com/issues/58274
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Impact of large PG splits

[Konstantin Shalygin]

Hi Eugene!

I have a case, where PG with millions of objects, like this

```
root@host# ./show_osd_pool_pg_usage.sh  | less | head
id  used_mbytes used_objects  omap_used_mbytes  omap_used_keys
--  ---     --
17.c91  1213.2482748031616  2539152   0 0
17.9ae  1213.3145303726196  2539025   0 0
17.1a4  1213.432228088379   2539752   0 0
17.8f4  1213.4958791732788  2539831   0 0
17.f9   1213.5339193344116  2539837   0 0
17.c9d  1213.564414024353   2540014   0 0
17.89   1213.6339054107666  2540183   0 0
17.412  1213.6393299102783  2539797   0 0
```

And OSD was very small, like 1TB with RocksDB ~150-200GB. Actually currently 
you see splitted PG. So one OSD was serve 64PG * 4M = 256,000,000 of objects...

Main problem was - to remove something, you need to move something. While the 
move is in progress, nothing is deleted
Also, deleting is slower than writing. So one task for all operations was 
impossible. I do it manually for a 9 moths. After the splitting of the some PG 
was completed, I took other PG away from the most crowded (from the operator’s 
point of view, problematic) OSD. The pgremapper [1] helped me with this. As far 
as I remember, in this way I got from 2048 to 3000 PG, then I was able to set 
4096 PG, after which it became possible to move to 4TV NVME

Your case doesn't look that scary. Firstly, your 85% means that you have 
hundreds of free gigabytes (8TB's). If new data does not arrive, the 
reservation mechanism is sufficient and after some time the process will end. 
On the other hand, I had a replica, so compared to the EC - my case is a simpler

In any case, it’s worth trying and using the maximum capabilities of the upmap


Good luck,
k

[1] https://github.com/digitalocean/pgremapper

> On 9 Apr 2024, at 11:39, Eugen Block  wrote:
> 
> I'm trying to estimate the possible impact when large PGs are splitted. 
> Here's one example of such a PG:
> 
> PG_STAT  OBJECTS  BYTES OMAP_BYTES*  OMAP_KEYS*  LOG   DISK_LOGUP
> 86.3ff277708  4144030984090   0  3092  3092
> [187,166,122,226,171,234,177,163,155,34,81,239,101,13,117,8,57,111]
> 
> Their main application is RGW on EC (currently 1024 PGs on 240 OSDs), 8TB 
> HDDs backed by SSDs. There are 6 RGWs running behind HAProxies. It took me a 
> while to convince them to do a PG split and now they're trying to assess how 
> big the impact could be. The fullest OSD is already at 85% usage, the least 
> filled one at 59%, so there is definitely room for a better balancing which, 
> will be necessary until the new hardware arrives. The current distribution is 
> around 100 PGs per OSD which usually would be fine, but since the PGs are 
> that large only a few PGs difference have a huge impact on the OSD 
> utilization.
> 
> I'm targeting 2048 PGs for that pool for now, probably do another split when 
> the new hardware has been integrated.
> Any comments are appreciated!

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph object gateway metrics

[Konstantin Shalygin]

Hi,

You can use the [2] exporter to achieve usage stats per user and per bucket, 
including quotas usage


k
Sent from my iPhone

> On 26 Mar 2024, at 01:38, Kushagr Gupta  wrote:
> 
> 2. https://github.com/blemmenes/radosgw_usage_exporter
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Laptop Losing Connectivity To CephFS On Sleep/Hibernation

[Konstantin Shalygin]

Hi,

Yes, this is generic solution for end users mounts - samba gateway


k
Sent from my iPhone

> On 23 Mar 2024, at 12:10, duluxoz  wrote:
> 
> Hi Alex, and thanks for getting back to me so quickly (I really appreciate 
> it),
> 
> So from what you said it looks like we've got the wrong solution. Instead, 
> (if I'm understanding things correctly) we may be better off setting up a 
> dedicated Samba server with the CephFS mounts, and then using Samba to share 
> those out - is that right?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW - tracking new bucket creation and bucket usage

[Konstantin Shalygin]

Hi,

> On 15 Mar 2024, at 01:07, Ondřej Kukla  wrote:
> 
> Hello I’m looking for suggestions how to track bucket creation over s3 api 
> and bucket usage (num of objects and size) of all buckets in time.
> 
> In our RGW setup, we have a custom client panel, where like 85% percent of 
> buckets are created which is easy for us to then track the newly created 
> buckets and assigned them to our clients, however we also enable them to 
> create the buckets using the s3 api and there the tracking bucket to client 
> is harder. We also want to periodically track the bucket usage stats (num of 
> objects and size) of all buckets in time.
> 
> Now we are using the GetBucketInfo from the AdminOPS api - 
> https://docs.ceph.com/en/quincy/radosgw/adminops/#id44 with the stats=true 
> option GET /admin/bucket?stats=1 which returns all buckets with the number of 
> objects and size we then parse. We also use it for the tracking of newly 
> created buckets as it gives you all of the buckets with their owners at one 
> place, but we are not using the metadata in that case.
> 
> This however in our environment, where we have close to ~400 buckets isn’t 
> scalable and the calls can take from 10 to 400 to even 6000s to complete.
> 
> So I’m looking for suggestions on how to get this data in time 
> better/smarter. I know there is a bunch of prometheus exporters like 
> https://github.com/pcuzner/rgw-exporter but they are usually just using the 
> same AdminOPS api so that doesn’t seem like a better solution.
> 
> I’m also working on sending all of our RGW ops logs to elastic which would 
> give me the ability to easily query the CreateBucket calls, but that only 
> solves the bucket creation and not the usage stats.
> 
> How would you handle this? Do you have any suggestions?


Do you try this [1] exporter? May be this code performance is better for you?


k
[1] https://github.com/blemmenes/radosgw_usage_exporter
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph metrics units

[Konstantin Shalygin]

Hi,

> On 14 Mar 2024, at 19:29, Denis Polom  wrote:
> 
> so metric itself is miliseconds and after division on _count it's in seconds?
> 
> 

This is two metrics for long running averages [1], the query that produces 
"seconds" unit looks like this

(irate(ceph_osd_op_r_latency_sum[1m]) / irate(ceph_osd_op_r_latency_count[1m])) 
* on (instance, ceph_daemon) group_left (device) ceph_disk_occupation * on 
(instance, ceph_daemon) group_left (device_class, hostname) ceph_osd_metadata


k
[1] 
https://docs.ceph.com/en/latest/mgr/prometheus/?highlight=%3Cname%3E_sum#statistic-names-and-labels

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph metrics units

[Konstantin Shalygin]

Hi,

> On 14 Mar 2024, at 16:44, Denis Polom  wrote:
> 
> do you know if there is some table of Ceph metrics and units that should be 
> used for them?
> 
> I currently struggling with
> 
> ceph_osd_op_r_latency_sum
> 
> ceph_osd_op_w_latency_sum
> 
> if they are in ms or seconds?
> 
> Any idea please?

This metric will be seconds, after division on _count metric


k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Telemetry endpoint down?

[Konstantin Shalygin]

Hi Greg

Seems is up now, last report uploaded successfully

Thanks,
k

Sent from my iPhone

> On 11 Mar 2024, at 18:57, Gregory Farnum  wrote:
> 
> We had a lab outage Thursday and it looks like this service wasn’t
> restarted after that occurred. Fixed now and we’ll look at how to prevent
> that in future.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Telemetry endpoint down?

[Konstantin Shalygin]

Hi, seems telemetry endpoint is down for a some days? We have connection errors 
from multiple places


1:ERROR Mar 10 00:46:10.653 [564383]: opensock: Could not establish a 
connection to telemetry.ceph.com:443
2:ERROR Mar 10 01:48:20.061 [564383]: opensock: Could not establish a 
connection to telemetry.ceph.com:443
3:ERROR Mar 10 02:50:29.473 [564383]: opensock: Could not establish a 
connection to telemetry.ceph.com:443
4:ERROR Mar 10 03:52:38.877 [564383]: opensock: Could not establish a 
connection to telemetry.ceph.com:443
5:ERROR Mar 10 04:54:48.285 [564383]: opensock: Could not establish a 
connection to telemetry.ceph.com:443
6:ERROR Mar 10 05:56:57.693 [564383]: opensock: Could not establish a 
connection to telemetry.ceph.com:443
7:ERROR Mar 10 06:59:07.105 [564383]: opensock: Could not establish a 
connection to telemetry.ceph.com:443
8:ERROR Mar 10 08:01:16.509 [564383]: opensock: Could not establish a 
connection to telemetry.ceph.com:443
9:ERROR Mar 10 09:03:25.917 [564383]: opensock: Could not establish a 
connection to telemetry.ceph.com:443 


Thanks,
k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Running dedicated RGWs for async tasks

[Konstantin Shalygin]

Hi,

Yes. You need to turn off gc, lc threads in config for your current (client 
side) RGW's.

Then setup your 'async tasks' RGW without client traffic. No special 
configuration needed, only if I wanna tune gc, lc settings


k
Sent from my iPhone

> On 7 Mar 2024, at 13:09, Marc Singer  wrote:
> 
> Hello Ceph Users
> 
> Since we are running a big S3 cluster we would like to externalize the RGW 
> daemons that do async tasks, like:
> 
> * Garbage collection
> * Lifecycle policies
> * Calculating and updating quotas
> 
> Would this be possible to do in the configuration? Which config values would 
> I need to set on the exposed RGWs and which on the Async Task RGWs?
> 
> Thanks for your input.
> 
> Marc
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph Quincy to Reef non cephadm upgrade

[Konstantin Shalygin]

Hi, 

Yes, you upgrade ceph-common package, then restart your mons

k
Sent from my iPhone

> On 6 Mar 2024, at 21:55, sarda.r...@gmail.com wrote:
> 
> My question is - does this mean I need to upgrade all ceph packages (ceph, 
> ceph-common) and restart only monitor daemon first?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Monitoring Ceph Bucket and overall ceph cluster remaining space

[Konstantin Shalygin]

Hi,

Don't aware about what is SW, but if this software works with Prometheus 
metrics format - why not. Anyway the exporters are open source, you can modify 
the existing code for your environment


k

Sent from my iPhone

> On 6 Mar 2024, at 07:58, Michael Worsham  wrote:
> 
> This looks interesting, but instead of Prometheus, could the data be exported 
> for SolarWinds?
> 
> The intent is to have SW watch the available storage space allocated and then 
> to alert when a certain threshold is reached (75% remaining for a warning; 
> 95% remaining for a critical).
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Monitoring Ceph Bucket and overall ceph cluster remaining space

[Konstantin Shalygin]

Hi, 

For RGW usage statistics you can use radosgw_usage_exporter [1]


k
[1] https://github.com/blemmenes/radosgw_usage_exporter

Sent from my iPhone

> On 6 Mar 2024, at 00:21, Michael Worsham  wrote:
> Is there an easy way to poll the ceph cluster buckets in a way to see how 
> much space is remaining? And is it possible to see how much ceph cluster 
> space is remaining overall? I am trying to extract the data from our  Ceph 
> cluster and put it into a format that our SolarWinds can understand in whole 
> number integers, so we can monitor bucket allocated space and overall cluster 
> space in the cluster as a whole.
> 
> Via Canonical support, the said I can do something like "sudo ceph df -f 
> json-pretty" to pull the information, but what is it I need to look at from 
> the output (see below) to display over to SolarWinds?
> 
> {
> "stats": {
> "total_bytes": 960027263238144,
> "total_avail_bytes": 403965214187520,
> "total_used_bytes": 556062049050624,
> "total_used_raw_bytes": 556062049050624,
> "total_used_raw_ratio": 0.57921481132507324,
> "num_osds": 48,
> "num_per_pool_osds": 48,
> "num_per_pool_omap_osds": 48
> },
> "stats_by_class": {
> "ssd": {
> "total_bytes": 960027263238144,
> "total_avail_bytes": 403965214187520,
> "total_used_bytes": 556062049050624,
> "total_used_raw_bytes": 556062049050624,
> "total_used_raw_ratio": 0.57921481132507324
> }
> },
> 
> And a couple of data pools...
> {
> "name": "default.rgw.jv-va-pool.data",
> "id": 65,
> "stats": {
> "stored": 4343441915904,
> "objects": 17466616,
> "kb_used": 12774490932,
> "bytes_used": 13081078714368,
> "percent_used": 0.053900588303804398,
> "max_avail": 76535973281792
> }
> },
> {
> "name": "default.rgw.jv-va-pool.index",
> "id": 66,
> "stats": {
> "stored": 42533675008,
> "objects": 401,
> "kb_used": 124610380,
> "bytes_used": 127601028363,
> "percent_used": 0.00055542576592415571,
> "max_avail": 76535973281792
> }
> },
> This message and its attachments are from Data Dimensions and are intended 
> only for the use of the individual or entity to which it is addressed, and 
> may contain information that is privileged, confidential, and exempt from 
> disclosure under applicable law. If the reader of this message is not the 
> intended recipient, or the employee or agent responsible for delivering the 
> message to the intended recipient, you are hereby notified that any 
> dissemination, distribution, or copying of this communication is strictly 
> prohibited. If you have received this communication in error, please notify 
> the sender immediately and permanently delete the original email and destroy 
> any copies or printouts of this email as well as any attachments.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Size return by df

[Konstantin Shalygin]

Hi,

Yes you can, this controlled by option

client quota df = false


k
Sent from my iPhone

> On Feb 22, 2024, at 11:17, Albert Shih  wrote:
> 
> Is they are any way to keep the first answer ?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: pacific 16.2.15 QE validation status

[Konstantin Shalygin]

> 
> On Feb 7, 2024, at 16:59, Zakhar Kirpichenko  wrote:
> 
> Indeed, it looks like it's been recently reopened. Thanks for this!

Hi,

It was merged yesterday

Thanks for the right noise,
k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph 16.2.14: ceph-mgr getting oom-killed

[Konstantin Shalygin]

Hi,

The backport to pacific was rejected [1], you may switch to reef, when [2] 
merged and released


[1] https://github.com/ceph/ceph/pull/55109
[2] https://github.com/ceph/ceph/pull/55110

k
Sent from my iPhone

> On Jan 25, 2024, at 04:12, changzhi tan <544463...@qq.com> wrote:
> 
> Is there any way to solve this problem？thanks
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Nautilous 14.2.22 slow OSD memory leak?

[Konstantin Shalygin]

Hi,

> On Jan 12, 2024, at 12:01, Frédéric Nass  
> wrote:
> 
> Hard to tell for sure since this bug hit different major versions of the 
> kernel, at least RHEL's from what I know. 

In what RH kernel release this issue was fixed?


Thanks,
k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS recovery with existing pools

[Konstantin Shalygin]

Good to hear that, Eugen!
CC'ed Zac for a your docs mention


k

> On Dec 11, 2023, at 23:28, Eugen Block  wrote:
> 
> Update: apparently, we did it!
> We walked through the disaster recovery steps where one of the steps was to 
> reset the journal. I was under the impression that the specified command 
> 'cephfs-journal-tool [--rank=N] journal reset' would simply reset all the 
> journals (mdlog and purge_queue), but it seems like it doesn't. After Mykola 
> (once again, thank you so much for your input) pointed towards running the 
> command for the purge_queue specifically, the filesystem got out of the 
> read-only mode and was mountable again. the exact command was:
> 
> cephfs-journal-tool --rank=cephfs:0 --journal=purge_queue journal reset
> 
> We didn't have to walk through the recovery with an empty pool, which is 
> nice. I have a suggestion to include the "journal inspect" command to the 
> docs for both mdlog and purge_queue to understand that both journals might 
> need a reset.
> 
> Thanks again, Mykola!
> Eugen

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: osdmaptool target & deviation calculation

[Konstantin Shalygin]

Hi,

This deviation is very soft. If u wanna do real upmaps you should use deviation 
1

k

Sent from my iPhone

> On Nov 27, 2023, at 21:39, Robert Hish  wrote:
> 
> The result is many many OSDs with a deviation well above the 
> upmap_max_deviation which is at default: 5
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CLT Meeting minutes 2023-11-23

[Konstantin Shalygin]

Hi,

> On Nov 23, 2023, at 16:10, Nizamudeen A  wrote:
> 
> RCs for reef, quincy and pacific
>   for next week when there is more time to discuss

Just little noise: pacific is ready? 16.2.15 should be last release (at least 
that was the last plan), but [1] still not merged. Why now ticket is closed - I 
don't know

Also many users reports about OOM in 16.2.14 release: patch also should be 
merged to main first [2]



Thanks,
k

[1] https://tracker.ceph.com/issues/62815
[2] https://tracker.ceph.com/issues/59580
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Bug fixes in 17.2.7

[Konstantin Shalygin]

Hi,

> On Nov 20, 2023, at 19:24, Tobias Kulschewski  
> wrote:
> 
> do you have a rough estimate of when this will happen?
> 
> 

Not at this year I think. For now precedence for a 18.2.1 and last release of 
Pacific
But you can request shaman build, and clone repo for your local usage


k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Bug fixes in 17.2.7

[Konstantin Shalygin]

Hi Tobias,

This was not meged to Quincy yet [1]

k

[1] https://tracker.ceph.com/issues/59730
Sent from my iPhone

> On Nov 20, 2023, at 17:50, Tobias Kulschewski  
> wrote:
> 
> Just wanted to ask, if the bug with the multipart upload [1] has been fixed 
> in 17.2.7?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: HDD cache

[Konstantin Shalygin]

Hi Peter,

> On Nov 8, 2023, at 20:32, Peter  wrote:
> 
> Anyone experienced this can advise?

You can try:

* check for current cache status

smartctl -x /dev/sda | grep "Write cache"

* turn off write cache

smartctl -s wcache-sct,off,p /dev/sda

* check again

smartctl -x /dev/sda | grep "Write cache"


Good luck,
k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Problem: Upgrading CEPH Pacific to Quincy resulted in CEPH Storage pool to stop functioning.

[Konstantin Shalygin]

Hi,

You need revert your packages from Quincy to Pacific. `dnf downgrade ceph-mon` 
command should help with this

k
Sent from my iPhone

> On Oct 11, 2023, at 03:22, Waywatcher  wrote:
> 
> I am unable to get any of the current monitors to run.  They all fail to start
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Problem: Upgrading CEPH Pacific to Quincy resulted in CEPH Storage pool to stop functioning.

[Konstantin Shalygin]

Hi,

For this upgrade you need at least some mon's up, then you can redeploy your 
pacific mon's to rocksdb

k
Sent from my iPhone

> On Oct 10, 2023, at 02:01, Waywatcher  wrote:
> 
> I upgraded my CEPH cluster without properly following the mon upgrade so
> they were no longer on leveldb.
> 
> Proxmox and CEPH were updated to the latest for the current release.
> https://pve.proxmox.com/wiki/Ceph_Pacific_to_Quincy
> 
>   1. The upgrade to Quincy states a recommendation that Mons are using
>   RocksDB.
>   2. Leveldb support has been removed from quincy.
> 
> 
> The monitors were still running as leveldb.
> 
>   1. Does this mean the mons cannot work at all since they are levelDB?
> 
> 
> I upgraded all nodes to the quincy release 17.2.6 and restarted the mons.
> 
> At this point the cluster stopped responding.
> `ceph` commands do not work since the service fails to start.
> 
> Are there steps for recovery?
> 
> 1) Roll back to Pacific without being able to use CEPH commands (ceph orch
> upgrade start --ceph-version ).
> 2) Rebuild the monitors using data from the OSDs while maintaining Quincy
> release.
> 3) Is this actually related to the bug about 17.2.6 (which is what
> Proxmox/CEPH upgrades to) https://tracker.ceph.com/issues/58156 ?
> 
> 
> I ran the upgrade on another cluster prior to this without issue. The Mons
> were set with RocksDB and running on Quincy 17.2.6.
> 
> I appreciate any suggestions.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: replacing storage server host (not drives)

[Konstantin Shalygin]

Hi,

The procedure is simple: get another host and put current disk to new host. 
Setup boot and network's and back to business


k
Sent from my iPhone

> On Sep 26, 2023, at 17:38, Wyll Ingersoll  
> wrote:
> 
> What is the recommended procedure for replacing the host itself without 
> destroying the OSDs or losing data?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs mount 'stalls'

[Konstantin Shalygin]

Hi,

William, this is fuse client, not the kernel

Mark, you can use kernel client. Stock c7 or install, for example, kernel-ml 
from ELrepo [1], and use the latest krbd version


[1] https://elrepo.org/tiki/HomePage#elrepo-kernel
k

Sent from my iPhone

> On 17 Sep 2023, at 22:10, William Edwards  wrote:
> What’s in dmesg?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Rocksdb compaction and OSD timeout

[Konstantin Shalygin]

Hi Igor,

> On 12 Sep 2023, at 15:28, Igor Fedotov  wrote:
> 
> Default hybrid allocator (as well as AVL one it's based on) could take 
> dramatically long time to allocate pretty large (hundreds of MBs) 64K-aligned 
> chunks for BlueFS. At the original cluster it was exposed as 20-30 sec OSD 
> stalls.

For the chunks, this mean bluestore min alloc size?
This cluster was deployed pre Pacific (64k) and not redeployed to Pacific 
default (4k)?


Thanks,
k
Sent from my iPhone

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Rocksdb compaction and OSD timeout

[Konstantin Shalygin]

This cluster use the default settings or something for Bluestore was changed?

You can check this via `ceph config diff`


As Mark said, it will be nice to have a tracker, if this really release problem

Thanks,
k
Sent from my iPhone

> On 7 Sep 2023, at 20:22, J-P Methot  wrote:
> 
> We went from 16.2.13 to 16.2.14
> 
> Also, timeout is 15 seconds because it's the default in Ceph. Basically, 15 
> seconds before Ceph shows a warning that OSD is timing out.
> 
> We may have found the solution, but it would be, in fact, related to 
> bluestore_allocator and not the compaction process. I'll post the actual 
> resolution when we confirm 100% that it works.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Rocksdb compaction and OSD timeout

[Konstantin Shalygin]

Hi,

> On 7 Sep 2023, at 18:21, J-P Methot  wrote:
> 
> Since my post, we've been speaking with a member of the Ceph dev team. He 
> did, at first, believe it was an issue linked to the common performance 
> degradation after huge deletes operation. So we did do offline compactions on 
> all our OSDs. It fixed nothing and we are going through the logs to try and 
> figure this out.
> 
> To answer your question, no the OSD doesn't restart after it logs the 
> timeout. It manages to get back online by itself, at the cost of sluggish 
> performances for the cluster and high iowait on VMs.
> 
> We mostly run RBD workloads.
> 
> Deep scrubs or no deep scrubs doesn't appear to change anything. Deactivating 
> scrubs altogether did not impact performances in any way.
> 
> Furthermore, I'll stress that this is only happening since we upgraded to the 
> latest Pacific, yesterday.

What is your previous release version? What is your OSD drives models?
The timeout are always 15s? Not 7s, not 17s?


Thanks,
k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Rocksdb compaction and OSD timeout

[Konstantin Shalygin]

Hi,

> On 7 Sep 2023, at 10:05, J-P Methot  wrote:
> 
> We're running latest Pacific on our production cluster and we've been seeing 
> the dreaded 'OSD::osd_op_tp thread 0x7f346aa64700' had timed out after 
> 15.00954s' error. We have reasons to believe this happens each time the 
> RocksDB compaction process is launched on an OSD. My question is, does the 
> cluster detecting that an OSD has timed out interrupt the compaction process? 
> This seems to be what's happening, but it's not immediately obvious. We are 
> currently facing an infinite loop of random OSDs timing out and if the 
> compaction process is interrupted without finishing, it may explain that.

You run the online compacting for this OSD's (`ceph osd compact ${osd_id}` 
command), right?



k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Status of diskprediction MGR module?

[Konstantin Shalygin]

Hi,

> On 28 Aug 2023, at 12:45, Robert Sander  wrote:
> 
> Several years ago the diskprediction module was added to the MGR collecting 
> SMART data from the OSDs.
> 
> There were local and cloud modes available claiming different accuracies. Now 
> only the local mode remains.
> 
> What is the current status of that MGR module (diskprediction_local)?
> 
> We have a cluster where SMART data is available from the disks (tested with 
> smartctl and visible in the Ceph dashboard), but even with an enabled 
> diskprediction_local module no health and lifetime info is shown.

The module don't have new commits for more than two year
I suggest to use smartctl_exporter [1] for monitoring drives health


[1] https://github.com/prometheus-community/smartctl_exporter
k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: User + Dev Monthly Meeting Minutes 2023-08-24

[Konstantin Shalygin]

On 24 Aug 2023, at 18:51, Laura Flores  wrote:
> 
> Need pacific backport for https://tracker.ceph.com/issues/59478
> 
>   - Laura will check on this, although a Pacific backport is unlikely due
>   to incompatibilities from the scrub backend refactoring.

Laura, this fix "for malformed fix" of earlier Pacific release or lack of this 
fix still prevents deleting snapshots created on previous release (example: 
created snapshot at Luminous, then upgrade to Nautilus->Pacific)?


Thanks,
k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [quincy] Migrating ceph cluster to new network, bind OSDs to multple public_nework

[Konstantin Shalygin]

Hi,

This how OSD's woks. For change the network subnet you need to setup 
reachability of both: old and new network, until end of migration

k
Sent from my iPhone

> On 22 Aug 2023, at 10:43, Boris Behrens  wrote:
> 
> The OSDs are still only bound to one IP address.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Check allocated RGW bucket/object size after enabling Bluestore compression

[Konstantin Shalygin]

Hi,

> On 17 Aug 2023, at 18:21, yosr.kchao...@gmail.com wrote:
> 
> Thanks for your reply. By the Bluestore compression I mean the compression 
> enabled on the pool level. It is also called inline compression.
> https://docs.ceph.com/en/reef/rados/configuration/bluestore-config-ref/#inline-compression
> 
> Do you see what I mean now ?

Yes, you can check metrics 'compress_bytes_used', 'compress_under_bytes', see 
[1]


[1] https://github.com/ceph/ceph/pull/37289
k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Check allocated RGW bucket/object size after enabling Bluestore compression

[Konstantin Shalygin]

Hi,

What you mean, Bluestore compression? The rgw compression is a rados 
compression, not the compress by rgw itself. You can setup different storage 
classes and upload to same pool uncompressed, or compressed objects

The compression ratio you can determine with exporter [1]

[1] https://github.com/blemmenes/radosgw_usage_exporter

k
Sent from my iPhone

> On 17 Aug 2023, at 17:58, yosr.kchao...@gmail.com wrote:
> I see that if we enable RGW compression, we can see the actual (before 
> compression) and utilized (after compression) size of the bucket.
> Is it possible to find the same information but with Bluestore compression ?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Can't join new mon - lossy channel, failing

[Konstantin Shalygin]

> On 16 Aug 2023, at 13:23, Josef Johansson  wrote:
> 
> I'm running ceph version 15.2.16 (a6b69e817d6c9e6f02d0a7ac3043ba9cdbda1bdf) 
> octopus (stable), that would mean I am not running the fix.
> 
> Glad to know that an upgrade will solve the issue!

I'm not 100% sure that this tracker, exactly fix exactly this [ipaddr deadlock 
"somewhere"] issue, but looks very similar


Thanks!
k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Can't join new mon - lossy channel, failing

[Konstantin Shalygin]

Hi,

> On 16 Aug 2023, at 11:30, Josef Johansson  wrote:
> 
> Let's do some serious necromancy here.
> 
> I just had this exact problem. Turns out that after rebooting all nodes (one 
> at the time of course), the monitor could join perfectly.
> 
> Why? You tell me. We did not see any traces of the ip address in any dumps 
> that we could get a hold of. I restarted all ceph-mgr beforehand as well.

What is your release?
This deadlock may be fixed via [1]


[1] https://tracker.ceph.com/issues/55355
k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: v18.2.0 Reef released

[Konstantin Shalygin]

Hi,

Thanks for the release!

Please, upgrade the OS Platform docs, currently there lack of Reef in ABC 
tests: https://tracker.ceph.com/issues/62354


Thanks,
k

> On 7 Aug 2023, at 21:37, Yuri Weinstein  wrote:
> 
> We're very happy to announce the first stable release of the Reef series.
> 
> We express our gratitude to all members of the Ceph community who
> contributed by proposing pull requests, testing this release,
> providing feedback, and offering valuable suggestions.
> 
> Major Changes from Quincy:
> - RADOS: RocksDB has been upgraded to version 7.9.2.
> - RADOS: There have been significant improvements to RocksDB iteration
> overhead and performance.
> - RADOS: The perf dump and perf schema commands have been deprecated
> in favor of the new counter dump and counter schema commands.
> - RADOS: Cache tiering is now deprecated.
> - RADOS: A new feature, the "read balancer", is now available, which
> allows users to balance primary PGs per pool on their clusters.
> - RGW: Bucket resharding is now supported for multi-site configurations.
> - RGW: There have been significant improvements to the stability and
> consistency of multi-site replication.
> - RGW: Compression is now supported for objects uploaded with
> Server-Side Encryption.
> - Dashboard: There is a new Dashboard page with improved layout.
> Active alerts and some important charts are now displayed inside
> cards.
> - RBD: Support for layered client-side encryption has been added.
> - Telemetry: Users can now opt in to participate in a leaderboard in
> the telemetry public dashboards.
> 
> We encourage you to read the full release notes at
> https://ceph.io/en/news/blog/2023/v18-2-0-reef-released/
> 
> Getting Ceph
> 
> * Git at git://github.com/ceph/ceph.git
> * Tarball at https://download.ceph.com/tarballs/ceph-18.2.0.tar.gz
> * Containers at https://quay.io/repository/ceph/ceph
> * For packages, see https://docs.ceph.com/docs/master/install/get-packages/
> * Release git sha1: 5dd24139a1eada541a3bc16b6941c5dde975e26d
> 
> Did you know? Every Ceph release is built and tested on resources
> funded directly by the non-profit Ceph Foundation.
> If you would like to support this and our other efforts, please
> consider joining now https://ceph.io/en/foundation/.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Quincy and liburing.so.2 on Rocky Linux 9

[Konstantin Shalygin]

Hi,

In most cases the 'Alternative' distro like Alma or Rocky have outdated 
versions of packages, if we compared it with CentOS Stream 8 or CentOS Stream 
9. For example is a golang package, on c8s is a 1.20 version on Alma still 1.19

You can try to use c8s/c9s or try to contribute to your distro to resolve 
dependency issues


k
Sent from my iPhone

> On 4 Aug 2023, at 02:05, dobr...@gmu.edu wrote:
> 
> I've been digging and I can't see that this has come up anywhere.
> 
> I'm trying to update a client from Pacific 17.2.3-2 to 17.2.6-4 and I'm 
> getting the error
> 
> Error: 
> Problem: cannot install the best update candidate for package 
> ceph-base-2:17.2.3-2.el9s.x86_64
>  - nothing provides liburing.so.2()(64bit) needed by 
> ceph-base-2:17.2.6-4.el9s.x86_64
>  - nothing provides liburing.so.2(LIBURING_2.0)(64bit) needed by 
> ceph-base-2:17.2.6-4.el9s.x86_64
> (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to 
> use not only best candidate packages)
> 
> Did Ceph Pacific switch to requiring liburing 2? Rocky 9 only provides 0.7-7. 
> CentOS stream seems to have 1.0.7-3 (at least back to when I set up that repo 
> on Foreman; I don't remember if I'm keeping it up-to-date).
> 
> Can I/should I just do --nobest when updating? I could probably build it from 
> a source RPM from another RH-based distro, but I'd rather keep it clean with 
> the same distro.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Luminous Bluestore issues and RGW Multi-site Recovery

[Konstantin Shalygin]

Hi,

Can you show `smartctl -a` for this device?
This drives show input/output errors in dmesg when you try to run ceph-osd?


k
Sent from my iPhone

> On 2 Aug 2023, at 21:44, Greg O'Neill  wrote:
> 
> Syslog says the drive is not in write-protect mode, however smart says life 
> remaining is at 1%.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Luminous Bluestore issues and RGW Multi-site Recovery

[Konstantin Shalygin]

Hi
> 
> On 31 Jul 2023, at 03:12, Gregory O'Neill  wrote:
> 
> /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0
> == "unexpected aio error")

The Kernel was returned error actually. Try look to dmesg, may be your SATA SSD 
in read only mode


k
Sent from my iPhone

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm and kernel memory usage

[Konstantin Shalygin]

Without determining what exactly process (kernel or userspace) "eat" memory, 
the ceph-users can't tell what exactly use memory, because don't see your 
display with your eyes 

You should run this commands on good & bad hosts to see the real difference. 
This may be related to kernel version, or Ceph options in container config or 
...


k
Sent from my iPhone

> On 26 Jul 2023, at 07:26, Luis Domingues  wrote:
> 
> First, thank you for taking time to reply to me.
> 
> However, my question was not on user-space memory neither on cache usage, as 
> I can see on my machines everything sums up quite nicely.
> 
> My question is: with packages, the non-cache kernel memory is around 2G to 
> 3G, while with Podman usage, it is more around 10G, and it can go up to 
> 40G-50G. Do anyone knows if this is expected and why this is the case?
> 
> Maybe this is a podman related question and ceph-dev is not the best place to 
> ask this kind of question, but maybe someone using cephadm saw similar 
> behavior.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm and kernel memory usage

[Konstantin Shalygin]

Good,

> On 24 Jul 2023, at 20:01, Luis Domingues  wrote:
> 
> Of course:
> 
> free -h
>  totalusedfree  shared  buff/cache   available
> Mem:  125Gi96Gi   9.8Gi   4.0Gi19Gi   
> 7.6Gi
> Swap:0B  0B  0B

As we can see, actually you have ~30GiB free (9.8GiB is not used & 19GiB is a 
page cache)
With this command you can determine what process actually use memory & how much

```
ps -eo size,pid,user,command | \
awk '{ hr=$1/1024 ; printf("%13.6f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { 
printf("%s ",$x) } print "" }' | \
sort -n

```

k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Does ceph permit the definition of new classes?

[Konstantin Shalygin]

Hi,

You definitely can add any other class name

k
Sent from my iPhone

> On 24 Jul 2023, at 16:04, wodel youchi  wrote:
> 
> Can I define new device classes in ceph, I know that there are hdd, ssd and
> nvme, but can I define other classes?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm and kernel memory usage

[Konstantin Shalygin]

Hi,

Can you paste `free -h` output for this hosts?


k
Sent from my iPhone

> On 24 Jul 2023, at 14:42, Luis Domingues  wrote:
> 
> Hi,
> 
> So after, looking into OSDs memory usage, which seem to be fine, on a 
> v16.2.13 running with cephadm, on el8, it seems that the kernel is using a 
> lot of memory.
> 
> # smem -t -w -k
> Area Used Cache Noncache
> firmware/hardware 0 0 0
> kernel image 0 0 0
> kernel dynamic memory 65.0G 18.6G 46.4G
> userspace memory 50.1G 260.5M 49.9G
> free memory 9.9G 9.9G 0
> -- 125.0G 28.8G 96.3G
> 
> Comparing with a similar other cluster, same OS, same ceph version, but 
> running packages instead if containers, and machines have a little bit more 
> memory:
> 
> # smem -t -w -k
> Area Used Cache Noncache
> firmware/hardware 0 0 0
> kernel image 0 0 0
> kernel dynamic memory 52.8G 50.5G 2.4G
> userspace memory 123.9G 198.5M 123.7G
> free memory 10.6G 10.6G 0
> -- 187.3G 61.3G 126.0G
> 
> Does anyone have an idea why when using containers with podman the kernel 
> needs a lot more memory?
> 
> Luis Domingues
> Proton AG
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: librbd hangs during large backfill

[Konstantin Shalygin]

Hi,

Check you libvirt limits for qemu open files/sockets. Seems, when you added new 
OSD's, your librbd client limit reached


k
Sent from my iPhone

> On 18 Jul 2023, at 19:32, Wesley Dillingham  wrote:
> 
> Did your automation / process allow for stalls in between changes to allow
> peering to complete? My hunch is you caused a very large peering storm
> (during peering a PG is inactive) which in turn caused your VMs to panic.
> If the RBDs are unmapped and re-mapped does it still continue to struggle?
> 
> Respectfully,
> 
> *Wes Dillingham*
> w...@wesdillingham.com
> LinkedIn 
> 
> 
>> On Tue, Jul 18, 2023 at 11:52 AM <
>> fb2cd0fc-933c-4cfe-b534-93d67045a...@simplelogin.com> wrote:
>> 
>> Starting on Friday, as part of adding a new pod of 12 servers, we
>> initiated a reweight on roughly 384 drives; from 0.1 to 0.25. Something
>> about the resulting large backfill is causing librbd to hang, requiring
>> server restarts. The volumes are showing buffer i/o errors when this
>> happens.We are currently using hybrid OSDs with both SSD and traditional
>> spinning disks. The current status of the cluster is:
>> ceph --version
>> ceph version 14.2.22
>> Cluster Kernel 5.4.49-200
>> {
>>"mon": {
>>"ceph version 14.2.22 nautilus (stable)": 3
>>},
>>"mgr": {
>>"ceph version 14.2.22 nautilus (stable)": 3
>>},
>>"osd": {
>>"ceph version 14.2.21 nautilus (stable)": 368,
>>"ceph version 14.2.22 (stable)": 2055
>>},
>>"mds": {},
>>"rgw": {
>>"ceph version 14.2.22 (stable)": 7
>>},
>>"overall": {
>>"ceph version 14.2.21 (stable)": 368,
>>"ceph version 14.2.22 (stable)": 2068
>>}
>> }
>> 
>> HEALTH_WARN, noscrub,nodeep-scrub flag(s) set.
>> pgs: 6815703/11016906121 objects degraded (0.062%) 2814059622/11016906121
>> objects misplaced (25.543%).
>> 
>> The client servers are on 3.10.0-1062.1.2.el7.x86_6
>> 
>> We have found a couple of issues that look relevant:
>> https://tracker.ceph.com/issues/19385
>> https://tracker.ceph.com/issues/18807
>> Has anyone experienced anything like this before? Does anyone have any
>> recommendations as to settings that can help alleviate this while the
>> backfill completes?
>> An example of the buffer ii/o errors:
>> 
>> Jul 17 06:36:08 host8098 kernel: buffer_io_error: 22 callbacks suppressed
>> Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical
>> block 0, async page read
>> Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical
>> block 0, async page read
>> Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical
>> block 0, async page read
>> Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical
>> block 0, async page read
>> Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical
>> block 0, async page read
>> Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical
>> block 0, async page read
>> Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical
>> block 3, async page read
>> Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-5, logical
>> block 511984, async page read
>> Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-6, logical
>> block 3487657728, async page read
>> Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-6, logical
>> block 3487657729, async page read
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Another Pacific point release?

[Konstantin Shalygin]

Hi,

> On 17 Jul 2023, at 12:53, Ponnuvel Palaniyappan  wrote:
> 
> The typical EOL date (2023-06-01) has already passed for Pacific. Just
> wondering if there's going to be another Pacific point release (16.2.14) in
> the pipeline.

Good point! At least, for possibility upgrade RBD clusters from Nautilus to 
Pacific, seems this release should get this backport [1]

Also, it will be good to see an update of information on distributions (ABC QA 
grades) [2]

Thanks,

[1] https://tracker.ceph.com/issues/59538
[2] https://docs.ceph.com/en/quincy/start/os-recommendations/#platforms

k


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MON sync time depends on outage duration

[Konstantin Shalygin]

This is a guess, the databases is like to swell. Especially the Level DB's, can 
grow x2 and reduce tens of percent of total size. This may be just another SST 
file creation, 1GB by default, Ii I remember it right

Do you was looks to Grafana, about this HDD's utilization, IOPS?

k
Sent from my iPhone

> On 7 Jul 2023, at 10:54, Eugen Block  wrote:
> 
> Can you share some more details what exactly you mean? Do you mean that if I 
> restart a MON three times it goes into compaction maintenance and that it's 
> not related to a timing? We tried the same on a different MON and only did 
> two tests:
> - stopping a MON for less than 5 minutes, starting it again, sync happens 
> immediately
> - stopping a MON for more than 5 minutes, starting it again, sync takes 15 
> minutes
> 
> This doesn't feel related to the payload size or keys option, but a timing 
> option.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MON sync time depends on outage duration

[Konstantin Shalygin]

Hi,

And additional to Dan suggestion, the HDD is not a good choices for RocksDB, 
which is most likely the reason for this thread, I think that from the 3rd time 
the database just goes into compaction maintenance


k
Sent from my iPhone

> On 6 Jul 2023, at 23:48, Eugen Block  wrote:
> The MON store is around 2 GB of size (on HDD)
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph quota qustion

[Konstantin Shalygin]

Hi,

This is incomplete multiparts I guess, you should remove it first. Don't know 
how S3 Browser works with this entities


k
Sent from my iPhone

> On 6 Jul 2023, at 07:57, sejun21@samsung.com wrote:
> 
> Hi, I contact you for some question about quota.
> 
> Situation is following below.
> 
> 1. I set the user quota 10M
> 2. Using s3 browser, upload one 12M file
> 3. The upload failed as i wish, but some object remains in the pool(almost 
> 10M) and s3brower doesn't show failed file.
> 
> I expected nothing to be left in Ceph. 
> 
> My question is "can user or admin remove the remaining objects?"
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Get bucket placement target

[Konstantin Shalygin]

Hi,

> On 3 Jul 2023, at 12:23, mahnoosh shahidi  wrote:
> 
> So clients can not get the value which they set in the LocationConstraint
> field in the create bucket request as in this doc
> ?

LocationConstraint in this case is a AZ [1], not the placement in Ceph (OSD 
pool, compression settings)


[1] https://docs.openstack.org/neutron/rocky/admin/config-az.html
k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Get bucket placement target

[Konstantin Shalygin]

Hi,

> On 2 Jul 2023, at 17:17, mahnoosh shahidi  wrote:
> 
> Is there any way for clients (without rgw-admin access) to get the
> placement target of their S3 buckets? The "GetBucketLocation'' api returns
> "default" for all placement targets and I couldn't find any other S3 api
> for this purpose.

From S3 side you can't know the internals of API. The client can operate with 
S3 STORAGE CLASS, for example the output

√ ~ % s5cmd ls s3://restic/snapshots/ | head
2023/06/30 17:10:03   263  
09c68acfe0cd536c3a21273a7adeee2911f370aa4f12fb9de5d13e1b8a93a7ef
2023/06/18 10:30:05   285  
0e6d2982310da04e9c003087457286f5526481facfcd24b617604353af6a00fb
2023/06/30 01:00:02   270  
133002c58d01afd187184bf4b25024d4247c173c01971f8f83409fb1bef8321a
2023/06/09 10:30:05   283  
18a75ad87f240ad3e26c337f0e2b5b43008153c2a3e525c99a3f5cca404ba369
2023/06/28 17:05:06   264  
19ad146ee7d6075d9450800f8b9bb920b30911c1812590409129eb5fcaa0aba5
2023/07/02 10:10:11   272  
1d3adb612e90d6e6eef88d9f2d0d496f231be7dc6befd1da870966da22b42a8a
2023/06/07 10:30:05   282  
1e676be243d7dd58bc39182ebb9767ffc8f8b9d49c8d812d343ed838fae76f4e
2023/06/05 01:00:03   268  
226adc2d95c43a5c88c894fa93a93f263e1ae80a31b40e4b6f1ce28d50c64979
2023/07/02 15:10:12   274  
2541bd2e646a78ab238675d8dc2eec6673cf4eb8354a7e6294e303c470facd07
2023/07/01 10:30:05   282  
28d272ef897c18a8baf545a426b48121a085e458dc78f76989200567ce05739d


You can add -s flag, to see the S3 STORAGE CLASS

√ ~ % s5cmd ls -s s3://restic/snapshots/ | head
2023/06/30 17:10:03 STANDARD263  
09c68acfe0cd536c3a21273a7adeee2911f370aa4f12fb9de5d13e1b8a93a7ef
2023/06/18 10:30:05 STANDARD285  
0e6d2982310da04e9c003087457286f5526481facfcd24b617604353af6a00fb
2023/06/30 01:00:02 STANDARD270  
133002c58d01afd187184bf4b25024d4247c173c01971f8f83409fb1bef8321a
2023/06/09 10:30:05 STANDARD283  
18a75ad87f240ad3e26c337f0e2b5b43008153c2a3e525c99a3f5cca404ba369
2023/06/28 17:05:06 STANDARD264  
19ad146ee7d6075d9450800f8b9bb920b30911c1812590409129eb5fcaa0aba5
2023/07/02 10:10:11 STANDARD272  
1d3adb612e90d6e6eef88d9f2d0d496f231be7dc6befd1da870966da22b42a8a
2023/06/07 10:30:05 STANDARD282  
1e676be243d7dd58bc39182ebb9767ffc8f8b9d49c8d812d343ed838fae76f4e
2023/06/05 01:00:03 STANDARD268  
226adc2d95c43a5c88c894fa93a93f263e1ae80a31b40e4b6f1ce28d50c64979
2023/07/02 15:10:12 STANDARD274  
2541bd2e646a78ab238675d8dc2eec6673cf4eb8354a7e6294e303c470facd07
2023/07/01 10:30:05 STANDARD282  
28d272ef897c18a8baf545a426b48121a085e458dc78f76989200567ce05739d

And, there we can see that S3 STORAGE CLASS is STANDARD


k


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Applying crush rule to existing live pool

[Konstantin Shalygin]

Hi,

> On 27 Jun 2023, at 18:31, Rok Jaklič  wrote:
> 
> I want to place an existing pool with data to ssd-s.
> 
> I've created crush rule:
> ceph osd crush rule create-replicated replicated_ssd default host ssd
> 
> If I apply this rule to the existing pool default.rgw.buckets.index with
> 180G of data with command:
> ceph osd pool set default.rgw.buckets.index crush_rule replicated_ssd
> 
> Will the rgw/cluster be available in the meantime while the cluster moves
> data?

Yes

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: rbd ls failed with operation not permitted

[Konstantin Shalygin]

Hi,

> On 7 Jun 2023, at 14:39, zyz  wrote:
> 
> When set the user's auth and then ls namespace, it is ok.
> 
> 
> But when I set the user's auth with namespace, ls namespace returns with 
> error, but why?

Because data with namespaces in "without namespace" space


k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: The pg_num from 1024 reduce to 32 spend much time, is there way to shorten the time？

[Konstantin Shalygin]

Hi,

> On 7 Jun 2023, at 10:02, Louis Koo  wrote:
> 
> I had set it from 0.05 to 1 with "ceph config set mon 
> target_max_misplaced_ratio 1.0", it's still invalid.


Because is setting for a mgr, not for mon, try `ceph config set mgr 
target_max_misplaced_ratio 1`

Cheers,
k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Duplicate help statements in Prometheus metrics in 16.2.13

[Konstantin Shalygin]

Hi Andreas,

> On 5 Jun 2023, at 14:57, Andreas Haupt  wrote:
> 
> after the update to CEPH 16.2.13 the Prometheus exporter is wrongly
> exporting multiple metric help & type lines for ceph_pg_objects_repaired:
> 
> [mon1] /root #curl -sS http://localhost:9283/metrics
> # HELP ceph_pg_objects_repaired Number of objects repaired in a pool Count
> # TYPE ceph_pg_objects_repaired counter
> ceph_pg_objects_repaired{poolid="34"} 0.0
> # HELP ceph_pg_objects_repaired Number of objects repaired in a pool Count
> # TYPE ceph_pg_objects_repaired counter
> ceph_pg_objects_repaired{poolid="33"} 0.0
> # HELP ceph_pg_objects_repaired Number of objects repaired in a pool Count
> # TYPE ceph_pg_objects_repaired counter
> ceph_pg_objects_repaired{poolid="32"} 0.0
> [...]
> 
> This annoys our exporter_exporter service so it rejects the export of ceph
> metrics. Is this a known issue? Will this be fixed in the next update?

We have backport for this issue [1]

[1] https://github.com/ceph/ceph/pull/51692

k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cluster without messenger v1, new MON still binds to port 6789

[Konstantin Shalygin]

Hi,

> On 1 Jun 2023, at 12:50, Robert Sander  wrote:
> 
> a cluster has ms_bind_msgr1 set to false in the config database.
> 
> Newly created MONs still listen on port 6789 and add themselves as providing 
> messenger v1 into the monmap.
> 
> How do I change that?
> 
> Shouldn't the MONs use the config for ms_bind_msgr1?

This config setting for listen, not for a "new mon born"
To disable msgr1 for mon completely, you should run command "ceph mon dump", 
and then use the v2 address and mon name as arg for a command, like this:

`ceph mon set-addrs mon1 v2:10.10.10.1:3300`

This will set only v2 address for your new mon


k

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: BlueStore fragmentation woes

[Konstantin Shalygin]

Hi Igor,

Should we backpot this to the p,q and reef release's?


Thanks,
k
Sent from my iPhone

> On 25 May 2023, at 23:13, Igor Fedotov  wrote:
> 
> You might be facing the issue fixed by https://github.com/ceph/ceph/pull/49885
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs max_file_size

[Konstantin Shalygin]

Hi,

> On 23 May 2023, at 13:27, Dietmar Rieder  wrote:
> 
> can the cephfs "max_file_size" setting be changed at any point in the 
> lifetime of a cephfs?
> Or is it critical for existing data if it is changed after some time? Is 
> there anything to consider when changing, let's say, from 1TB (default) to 
> 4TB ?
> 
> We are running the latest Nautilus release, BTW.

It's safe to set this option for any time. We have one project where 
max_file_size set to 7036874240 - everything works as expected


k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Dedicated radosgw gateways

[Konstantin Shalygin]

Hi,

> On 18 May 2023, at 23:04, Rok Jaklič  wrote:
> 
> I've searched for rgw_enable_lc_threads and rgw_enable_gc_threads a bit.
> 
> but there is little information about those settings. Is there any
> documentation in the wild about those settings?

This is Life Cycle (see S3 lifecycle policy documentation) and Garbage 
Collector (actual delete objects from OSD)

> 
> Are they enabled by default?

Yes


k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Discussion thread for Known Pacific Performance Regressions

[Konstantin Shalygin]

Hi Mark!

Thank you very much for this message, acknowledging the problem publicly is the 
beginning of fixing it ❤️

> On 11 May 2023, at 17:38, Mark Nelson  wrote:
> 
> Hi Everyone,
> 
> This email was originally posted to d...@ceph.io, but Marc mentioned that he 
> thought this would be useful to post on the user list so I'm re-posting here 
> as well.

Is there any plan/way to the fix performance regressions the current even 
release (e.g. 16), which can work no worse than release 14? The 16.2.14 release 
should fix the last issues that block updates (such as the inability to delete 
old snapshots). I'm concern because of the Ceph release cycle, which obliges 
the old release to EOL. The solution at least in the form "you need to redeploy 
all our OSD's", due impossibility RocksDB version migrations - seems okay. This 
is important for the development of internal products (based on RADOS as 
backend), whether it is worth putting into the team roadmaps the writing of 
"migrators from one cluster to another" on application level or, in principle, 
preparing to live forever on version 14 and, accordingly, take into account its 
scaling limits


Thanks,
k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Dedicated radosgw gateways

[Konstantin Shalygin]

Hi,

> On 15 May 2023, at 14:58, Michal Strnad  wrote:
> 
> at Cephalocon 2023, it was mentioned several times that for service tasks 
> such as data deletion via garbage collection or data replication in S3 via 
> zoning, it is good to do them on dedicated radosgw gateways and not mix them 
> with gateways used by users. How can this be achieved? How can we isolate 
> these tasks? Will using dedicated keyrings instead of admin keys be 
> sufficient? How do you operate this in your environment?

Just:

# don't put client traffic to "dedicated radosgw gateways"
# disable lc/gc on "gateways used by users" via `rgw_enable_lc_threads = false` 
& `rgw_enable_gc_threads = false`


k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CEPH Version choice

[Konstantin Shalygin]

Hi,

> On 15 May 2023, at 11:37, Tino Todino  wrote:
> 
> What are the main reasons for not upgrading to the latest and greatest?

One of the main reasons - "just can't", because your Ceph-based products will 
get worse at real (not benchmark) performance, see [1]


[1] 
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/2E67NW6BEAVITL4WTAAU3DFLW7LJX477/


k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ovirt integration with Ceph

[Konstantin Shalygin]

Hi,

Can you see logs at the vdsm.log file? What exactly happened on storage domain 
connection?


k
Sent from my iPhone

> On 26 Apr 2023, at 00:37, kushagra.gu...@hsc.com wrote:
> 
> Hi Team,
> 
> We are trying to integrate ceph with ovirt.
> We have deployed ovirt 4.4.
> We want to create a storage domain of POSIX compliant type for mounting a 
> ceph based infrastructure in ovirt.
> We have done SRV based resolution in our DNS server for ceph mon nodes but we 
> are unable to create a storage domain using that.
> 
> We are able to manually mount the ceph-mon nodes using the following command 
> on the deployment hosts:
> 
> sudo mount -t ceph :/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9 
> /rhev/data-center/mnt/:_volumes_xyz_conf_00593e1d-b674-4b00-a289-20bec06761c9 
> -o rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA==
> 
> [root@deployment-host mnt]# df -kh
> df: /run/user/0/gvfs: Transport endpoint is not connected
> Filesystem
>  Size  Used Avail 
> Use% Mounted on
> 
> [abcd:abcd:abcd::51]:6789,[abcd:abcd:abcd::52]:6789,[abcd:abcd:abcd::53]:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9
>19G 0   19G   0% 
> /rhev/data-center/mnt/:_volumes_xyz_conf_00593e1d-b674-4b00-a289-20bec06761c9
> 
> 
> Query:
> 1. Could anyone help us out with storage domain creation in ovirt for SRV 
> resolved ceph-mon nodes.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: deploying Ceph using FQDN for MON / MDS Services

[Konstantin Shalygin]

Hi,

Just add POSIX domain, fstype ceph
This is equivalent of mount -t Ceph on ovirt side


k
Sent from my iPhone

> On 21 Apr 2023, at 05:24, Lokendra Rathour  wrote:
> 
> Hi Robert / Team,
> Further we are now trying to integrate the ceph as storage domain in OVirt
> 4.4
> 
> 
> We want to create a storage domain of POSIX-compliant type for mounting a
> ceph-based infrastructure in oVirt.
> As stated we are able to manually mount the ceph-mon nodes using the
> following command on the oVirt deployment hosts:
> 
> sudo mount -t ceph :/volumes/xyz/conf/00593e1d-b674-4b00-a289-20becr06761c9
> /rhev/data-center/mnt/:_volumes_xyz_conf_00593e1d-b674-4b00-a289-20bec0r6761c9
> -o rw,name=foo,secret=AQABDzRkTar*Lnx6qX/VDA==
> 
> # mount on Node:
> 
> *[root@deployment-host mnt]# df -kh*
> 
> * df: /run/user/0/gvfs: Transport endpoint is not connected*
> 
> *
> Filesystem
> Size  Used Avail Use% Mounted on*
> 
> [abcd:abcd:abcd::51]:6789,[abcd:abcd:abcd::52]:6789,[abcd:abcd:abcd::53]:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9
> 19G 0   19G   0%
> /rhev/data-center/mnt/:_volumes_xyz_conf_00593e1d-b674-4b00-a289-20bec06761c9
> 
> 
> 
> Query:
> 1. Could anyone help us out with storage domain creation in oVirt, we need
> to ensure that Domain is always up and connected in the state of Active
> Monitor failure.
> 
>> On Tue, Apr 18, 2023 at 2:41 PM Lokendra Rathour 
>> wrote:
>> 
>> yes thanks, Robert,
>> after installing the Ceph common mount is working fine.
>> 
>> 
>> On Tue, Apr 18, 2023 at 2:10 PM Robert Sander <
>> r.san...@heinlein-support.de> wrote:
>> 
 On 18.04.23 06:12, Lokendra Rathour wrote:
>>> 
 but if I try mounting from a normal Linux machine with connectivity
 enabled between Ceph mon nodes, it gives the error as stated before.
>>> 
>>> Have you installed ceph-common on the "normal Linux machine"?
>>> 
>>> Regards
>>> --
>>> Robert Sander
>>> Heinlein Support GmbH
>>> Linux: Akademie - Support - Hosting
>>> http://www.heinlein-support.de
>>> 
>>> Tel: 030-405051-43
>>> Fax: 030-405051-19
>>> 
>>> Zwangsangaben lt. §35a GmbHG:
>>> HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
>>> Geschäftsführer: Peer Heinlein  -- Sitz: Berlin
>>> ___
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>> 
>> 
>> 
>> --
>> ~ Lokendra
>> skype: lokendrarathour
>> 
>> 
>> 
> 
> -- 
> ~ Lokendra
> skype: lokendrarathour
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: pacific el7 rpms

[Konstantin Shalygin]

On EL7 only Nautilus was present. Pacific was from EL8


k

> On 17 Apr 2023, at 11:29, Marc  wrote:
> 
> 
> Is there ever going to be rpms in 
> 
> https://download.ceph.com/rpm-pacific/el7/

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: deploying Ceph using FQDN for MON / MDS Services

[Konstantin Shalygin]

Hi,

This by the reason of DNS. Something from userland should be provide IP 
addresses for kernel


k
Sent from my iPhone

> On 17 Apr 2023, at 05:56, Lokendra Rathour  wrote:
> 
> Hi Team,
> The mount at the client side should be independent of Ceph, but here in
> this case of DNS SRV-based mount, we see that the Ceph common utility is
> needed.
> What can be the reason for the same, any inputs in this direction would be
> helpful.
> 
> Best Regards,
> Lokendra
> 
> 
>> On Sun, Apr 16, 2023 at 10:11 AM Lokendra Rathour 
>> wrote:
>> 
>> Hi .
>> Any input will be of great help.
>> Thanks once again.
>> Lokendra
>> 
>> On Fri, 14 Apr, 2023, 3:47 pm Lokendra Rathour, 
>> wrote:
>> 
>>> Hi Team,
>>> their is one additional observation.
>>> Mount as the client is working fine from one of the Ceph nodes.
>>> Command *: sudo mount -t ceph :/ /mnt/imgs  -o
>>> name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwdfULnx6qX/VDA== *
>>> 
>>> *we are not passing the Monitor address, instead, DNS SRV is configured
>>> as per:*
>>> https://docs.ceph.com/en/quincy/rados/configuration/mon-lookup-dns/
>>> 
>>> mount works fine in this case.
>>> 
>>> 
>>> 
>>> But if we try to mount from the other Location i.e from another
>>> VM/client(non-Ceph Node)
>>> we are getting the error :
>>>  mount -t  ceph :/ /mnt/imgs  -o
>>> name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA== -v
>>> *mount: /mnt/image: mount point does not exist.*
>>> 
>>> the document says that if we do not pass the monitor address, it tries
>>> discovering the monitor address from DNS Servers, but in actual it is not
>>> happening.
>>> 
>>> 
>>> 
>>> On Tue, Apr 11, 2023 at 6:48 PM Lokendra Rathour <
>>> lokendrarath...@gmail.com> wrote:
>>> 
 Ceph version Quincy.
 
 But now I am able to resolve the issue.
 
 During mount i will not pass any monitor details, it will be
 auto-discovered via SRV.
 
 On Tue, Apr 11, 2023 at 6:09 PM Eugen Block  wrote:
 
> What ceph version is this? Could it be this bug [1]? Although the
> error message is different, not sure if it could be the same issue,
> and I don't have anything to test ipv6 with.
> 
> [1] https://tracker.ceph.com/issues/47300
> 
> Zitat von Lokendra Rathour :
> 
>> Hi All,
>> Requesting any inputs around the issue raised.
>> 
>> Best Regards,
>> Lokendra
>> 
>> On Tue, 24 Jan, 2023, 7:32 pm Lokendra Rathour, <
> lokendrarath...@gmail.com>
>> wrote:
>> 
>>> Hi Team,
>>> 
>>> 
>>> 
>>> We have a ceph cluster with 3 storage nodes:
>>> 
>>> 1. storagenode1 - abcd:abcd:abcd::21
>>> 
>>> 2. storagenode2 - abcd:abcd:abcd::22
>>> 
>>> 3. storagenode3 - abcd:abcd:abcd::23
>>> 
>>> 
>>> 
>>> The requirement is to mount ceph using the domain name of MON node:
>>> 
>>> Note: we resolved the domain name via DNS server.
>>> 
>>> 
>>> For this we are using the command:
>>> 
>>> ```
>>> 
>>> mount -t ceph [storagenode.storage.com]:6789:/  /backup -o
>>> name=admin,secret=AQCM+8hjqzuZEhAAcuQc+onNKReq7MV+ykFirg==
>>> 
>>> ```
>>> 
>>> 
>>> 
>>> We are getting the following logs in /var/log/messages:
>>> 
>>> ```
>>> 
>>> Jan 24 17:23:17 localhost kernel: libceph: resolve '
>>> storagenode.storage.com' (ret=-3): failed
>>> 
>>> Jan 24 17:23:17 localhost kernel: libceph: parse_ips bad ip '
>>> storagenode.storage.com:6789'
>>> 
>>> ```
>>> 
>>> 
>>> 
>>> We also tried mounting ceph storage using IP of MON which is working
> fine.
>>> 
>>> 
>>> 
>>> Query:
>>> 
>>> 
>>> Could you please help us out with how we can mount ceph using FQDN.
>>> 
>>> 
>>> 
>>> My /etc/ceph/ceph.conf is as follows:
>>> 
>>> [global]
>>> 
>>> ms bind ipv6 = true
>>> 
>>> ms bind ipv4 = false
>>> 
>>> mon initial members = storagenode1,storagenode2,storagenode3
>>> 
>>> osd pool default crush rule = -1
>>> 
>>> fsid = 7969b8a3-1df7-4eae-8ccf-2e5794de87fe
>>> 
>>> mon host =
>>> 
> [v2:[abcd:abcd:abcd::21]:3300,v1:[abcd:abcd:abcd::21]:6789],[v2:[abcd:abcd:abcd::22]:3300,v1:[abcd:abcd:abcd::22]:6789],[v2:[abcd:abcd:abcd::23]:3300,v1:[abcd:abcd:abcd::23]:6789]
>>> 
>>> public network = abcd:abcd:abcd::/64
>>> 
>>> cluster network = eff0:eff0:eff0::/64
>>> 
>>> 
>>> 
>>> [osd]
>>> 
>>> osd memory target = 4294967296
>>> 
>>> 
>>> 
>>> [client.rgw.storagenode1.rgw0]
>>> 
>>> host = storagenode1
>>> 
>>> keyring = /var/lib/ceph/radosgw/ceph-rgw.storagenode1.rgw0/keyring
>>> 
>>> log file = /var/log/ceph/ceph-rgw-storagenode1.rgw0.log
>>> 
>>> rgw frontends = beast endpoint=[abcd:abcd:abcd::21]:8080
>>> 
>>> rgw thread pool size = 512
>>> 
>>> --
>>> ~ 

[ceph-users] Re: OSDs remain not in after update to v17

[Konstantin Shalygin]

Hi,

This PR for the main branch and was never backpoted to another branches, 
currently


k
Sent from my iPhone

> On 15 Apr 2023, at 21:00, Alexandre Becholey  wrote:
> 
> Hi,
> 
> Thank you for your answer, yes this seems to be exactly my issue. The pull 
> request related to the issue is this one: 
> https://github.com/ceph/ceph/pull/49199 and it is not (yet?) merged into the 
> Quincy release. Hopefully this will happen before the next major release, 
> because I cannot run any `ceph orch` command as they hang.
> 
> Kind regards,
> Alexandre
> 
> 
> --- Original Message ---
>> On Saturday, April 15th, 2023 at 6:26 PM, Ramin Najjarbashi 
>>  wrote:
>> 
>> 
>> Hi
>> I think the issue you are experiencing may be related to a bug that has been 
>> reported in the Ceph project. Specifically, the issue is documented in 
>> https://tracker.ceph.com/issues/58156, and a pull request has been submitted 
>> and merged in https://github.com/ceph/ceph/pull/44090.
>> 
>>> On Fri, Apr 14, 2023 at 8:17 PM Alexandre Becholey  wrote:
>>> 
>>> Dear Ceph Users,
>>> 
>>> I have a small ceph cluster for VMs on my local machine. It used to be 
>>> installed with the system packages and I migrated it to docker following 
>>> the documentation. It worked OK until I migrated from v16 to v17 a few 
>>> months ago. Now the OSDs remain "not in" as shown in the status:
>>> 
>>> # ceph -s
>>> cluster:
>>> id: abef2e91-cd07-4359-b457-f0f8dc753dfa
>>> health: HEALTH_WARN
>>> 6 stray daemon(s) not managed by cephadm
>>> 1 stray host(s) with 6 daemon(s) not managed by cephadm
>>> 2 devices (4 osds) down
>>> 4 osds down
>>> 1 host (4 osds) down
>>> 1 root (4 osds) down
>>> Reduced data availability: 129 pgs inactive
>>> 
>>> services:
>>> mon: 1 daemons, quorum bjorn (age 8m)
>>> mgr: bjorn(active, since 8m)
>>> osd: 4 osds: 0 up (since 4w), 4 in (since 4w)
>>> 
>>> data:
>>> pools: 2 pools, 129 pgs
>>> objects: 0 objects, 0 B
>>> usage: 1.8 TiB used, 1.8 TiB / 3.6 TiB avail
>>> pgs: 100.000% pgs unknown
>>> 129 unknown
>>> 
>>> I can see some network communication between the OSDs and the monitor and 
>>> the OSDs are running:
>>> 
>>> # docker ps -a
>>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
>>> f8fbe8177a63 quay.io/ceph/ceph:v17 "/usr/bin/ceph-osd -…" 9 minutes ago Up 
>>> 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-osd-2
>>> 6768ec871404 quay.io/ceph/ceph:v17 "/usr/bin/ceph-osd -…" 9 minutes ago Up 
>>> 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-osd-1
>>> ff82f84504d5 quay.io/ceph/ceph:v17 "/usr/bin/ceph-osd -…" 9 minutes ago Up 
>>> 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-osd-0
>>> 4c89e50ce974 quay.io/ceph/ceph:v17 "/usr/bin/ceph-osd -…" 9 minutes ago Up 
>>> 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-osd-3
>>> fe0b6089edda quay.io/ceph/ceph:v17 "/usr/bin/ceph-mon -…" 9 minutes ago Up 
>>> 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-mon-bjorn
>>> f76ac9dcdd6d quay.io/ceph/ceph:v17 "/usr/bin/ceph-mgr -…" 9 minutes ago Up 
>>> 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-mgr-bjorn
>>> 
>>> However when I try to use any `ceph orch` commands, they hang. I can also 
>>> see some blacklist on the OSDs:
>>> 
>>> # ceph osd blocklist ls
>>> 10.99.0.13:6833/3770763474 2023-04-13T08:17:38.885128+
>>> 10.99.0.13:6832/3770763474 2023-04-13T08:17:38.885128+
>>> 10.99.0.13:0/2634718754 2023-04-13T08:17:38.885128+
>>> 10.99.0.13:0/1103315748 2023-04-13T08:17:38.885128+
>>> listed 4 entries
>>> 
>>> The first two entries correspond to the manager process. `ceph osd 
>>> blocked-by` does not show anything.
>>> 
>>> I think I might have forgotten to set the `ceph osd require-osd-release 
>>> ...` because 14 is written in 
>>> `/var/lib/ceph//osd.?/require_osd_release`. If I try to do it now, the 
>>> monitor hits an abort:
>>> 
>>> debug 0> 2023-04-12T08:43:27.788+ 7f0fcf2aa700 -1 *** Caught signal 
>>> (Aborted) **
>>> in thread 7f0fcf2aa700 thread_name:ms_dispatch
>>> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy 
>>> (stable)
>>> 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f0fd94bbcf0]
>>> 2: gsignal()
>>> 3: abort()
>>> 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
>>> const*)+0x18f) [0x7f0fdb5124e3]
>>> 5: /usr/lib64/ceph/libceph-common.so.2(+0x26a64f) [0x7f0fdb51264f]
>>> 6: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, 
>>> std::map, 
>>> std::allocator >, boost::variant>> 7: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x38d) 
>>> [0x562719cb127d]
>>> 8: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x17b) 
>>> [0x562719cb18cb]
>>> 9: (PaxosService::dispatch(boost::intrusive_ptr)+0x2ce) 
>>> [0x562719c20ade]
>>> 10: (Monitor::handle_command(boost::intrusive_ptr)+0x1ebb) 
>>> [0x562719ab9f6b]
>>> 11: (Monitor::dispatch_op(boost::intrusive_ptr)+0x9f2) 
>>> [0x562719abe152]
>>> 12: (Monitor::_ms_dispatch(Message*)+0x406) [0x562719abf066]
>>> 13: 

[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...

[Konstantin Shalygin]

Hi,

Current controller mode is RAID. You can switch to HBA mode and disable cache 
in controller settings at the BIOS


k
Sent from my iPhone

> On 15 Apr 2023, at 12:11, Marco Gaiarin  wrote:
> 
> Mandi! Frank Schilder
>  In chel di` si favelave...
> 
>>>  iops: min=2, max=   40, avg=21.13, stdev= 6.10, samples=929
>>>  iops: min=2, max=   42, avg=21.52, stdev= 6.56, samples=926
>> That looks horrible.
> 
> Exactly, horrible.
> 
> The strange thing is that we came from an homegrown Ceph cluster built using
> old hardware (HP G6 servers) and spare disks, that perform 'better', or at
> least perform 'more uniformly' that this 'new'.
> 
> With these embarassing IOPS, sooner or later we reach the point where
> performances go to the ground, and sometimes it suffices to launch some
> 'find' on the filesystem involved...
> 
> 
>> We also have a few SATA HDDs in Dell servers and they do about 100-150 IOP/s 
>> read or write. Originally, I was also a bit afraid that these disks would 
>> drag performance down, but they are on par with the NL-SAS drives.
>> For ceph we use the cheapest Dell disk controller one can get (Dell HBA330 
>> Mini (Embedded)) and it works perfectly. All ceph-disks are configured 
>> non-raid, which is equivalent to JBOD mode or pass-through. These 
>> controllers have no cache options, if your do, disable all of them. Mode 
>> should be write-through.
>> For your disk type I saw "volatile write cache available = yes" on "the 
>> internet". This looks a bit odd, but maybe these HDDs do have some volatile 
>> cache. Try to disable it with smartctl and do the benchmark again.
> 
> Sorry, i'm a bit puzzled here.
> 
> Matthias suggest to enable write cache, you suggest to disble it... or i'm
> cache-confused?! ;-)
> 
> 
> My actually controller configuration is:
> 
> root@pppve1:~# perccli /c0 show
> Generating detailed summary of the adapter, it may take a while to complete.
> 
> CLI Version = 007.1910.. Oct 08, 2021
> Operating system = Linux 5.4.203-1-pve
> Controller = 0
> Status = Success
> Description = None
> 
> Product Name = PERC H750 Adapter
> Serial Number = 23L01Y6
> SAS Address =  5f4ee0802ba3a400
> PCI Address = 00:b3:00:00
> System Time = 04/14/2023 18:03:24
> Mfg. Date = 03/25/22
> Controller Time = 04/14/2023 16:03:22
> FW Package Build = 52.16.1-4405
> BIOS Version = 7.16.00.0_0x07100501
> FW Version = 5.160.02-3552
> Driver Name = megaraid_sas
> Driver Version = 07.713.01.00-rc1
> Current Personality = RAID-Mode 
> Vendor Id = 0x1000
> Device Id = 0x10E2
> SubVendor Id = 0x1028
> SubDevice Id = 0x2176
> Host Interface = PCI-E
> Device Interface = SAS-12G
> Bus Number = 179
> Device Number = 0
> Function Number = 0
> Domain ID = 0
> Security Protocol = None
> JBOD Drives = 6
> 
> JBOD LIST :
> =
> 
> -
> ID EID:Slt DID State Intf Med   Size SeSz ModelVendor   
> Port 
> -
> 0 64:0  6 Onln  SATA SSD 447.130 GB 512B MTFDDAK480TDTATA  x1 
>   
> 1 64:1  8 Onln  SATA SSD 447.130 GB 512B MTFDDAK480TDTATA  x1 
>   
> 3 64:3  7 Onln  SATA SSD 447.130 GB 512B MZ7KH480HAHQ0D3  ATA  x1 
>   
> 5 64:5  9 Onln  SATA HDD   3.638 TB 512B HGST HUS726T4TALA6L0 ATA  x1 
>   
> 6 64:6 10 Onln  SATA HDD   3.638 TB 512B HGST HUS726T4TALA6L0 ATA  x1 
>   
> 7 64:7 11 Onln  SATA HDD   3.638 TB 512B HGST HUS726T4TALA6L0 ATA  x1 
>   
> -
> 
> ID=JBOD Target ID|EID=Enclosure Device ID|Slt=Slot No|DID=Device 
> ID|Onln=Online
> Offln=Offline|Intf=Interface|Med=Media Type|SeSz=Sector Size
> SED=Self Encryptive Drive|PI=Protection Info|Sp=Spun|U=Up|D=Down
> 
> Physical Drives = 6
> 
> PD LIST :
> ===
> 
> --
> EID:Slt DID State DG   Size Intf Med SED PI SeSz ModelSp 
> Type 
> --
> 64:0  6 Onln  -  447.130 GB SATA SSD N   N  512B MTFDDAK480TDTU  
> JBOD 
> 64:1  8 Onln  -  447.130 GB SATA SSD N   N  512B MTFDDAK480TDTU  
> JBOD 
> 64:3  7 Onln  -  447.130 GB SATA SSD N   N  512B MZ7KH480HAHQ0D3  U  
> JBOD 
> 64:5  9 Onln  -3.638 TB SATA HDD N   N  512B HGST HUS726T4TALA6L0 U  
> JBOD 
> 64:6 10 Onln  -3.638 TB SATA HDD N   N  512B HGST HUS726T4TALA6L0 U  
> JBOD 
> 64:7 11 Onln  -3.638 TB SATA HDD N   N  512B HGST HUS726T4TALA6L0 U  
> JBOD 
> --
> 
> EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|DG=DriveGroup
> DHS=Dedicated Hot Spare|UGood=Unconfigured Good|GHS=Global Hotspare
> UBad=Unconfigured 

[ceph-users] Re: monitoring apply_latency / commit_latency ?

[Konstantin Shalygin]


Hi,

> On 2 Apr 2023, at 23:14, Matthias Ferdinand  wrote:
> 
> I understand that grafana graphs are generated from prometheus metrics.
> I just wanted to know which OSD daemon-perf values feed these prometheus
> metrics (or if they are generated in some other way).

Yep, this perf metrics is generated in some way 
You can consult with ceph-mgr prometheus module source code [1]


[1] 
https://github.com/ceph/ceph/blob/main/src/pybind/mgr/prometheus/module.py#L1656-L1676
k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: monitoring apply_latency / commit_latency ?

[Konstantin Shalygin]

Hi,

> On 25 Mar 2023, at 23:15, Matthias Ferdinand  wrote:
> 
> from "ceph daemon osd.X perf dump"?


No, from ceph-mgr prometheus exporter
You can enable it via `ceph mgr module enable prometheus`

> Please bear with me :-) I just try to get some rough understanding what
> the numbers to be collected and graphed actually mean and how they are
> related to each other.

I think you can find metrics descriptions at source of official Grafana 
dashborad [1]


[1] 
https://github.com/ceph/ceph/blob/main/monitoring/ceph-mixin/dashboards_out/osds-overview.json
k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: monitoring apply_latency / commit_latency ?

[Konstantin Shalygin]

Hi Matthias,

Prometheus exporter already have all this metrics, you can setup Grafana panels 
as you want
Also, the apply latency in a metric for a pre-bluestore, i.e. filestore
For Bluestore apply latency is the same as commit latency, you can check this 
via `ceph osd perf` command




k

> On 25 Mar 2023, at 00:02, Matthias Ferdinand  wrote:
> 
> Hi,
> 
> I would like to understand how the per-OSD data from "ceph osd perf"
> (i.e.  apply_latency, commit_latency) is generated. So far I couldn't
> find documentation on this. "ceph osd perf" output is nice for a quick
> glimpse, but is not very well suited for graphing. Output values are
> from the most recent 5s-averages apparently.
> 
> With "ceph daemon osd.X perf dump" OTOH, you get quite a lot of latency
> metrics, while it is just not obvious to me how they aggregate into
> apply_latency and commit_latency. Or some comparably easy read latency
> metric (something that is missing completely in "ceph osd perf").
> 
> Can somebody shed some light on this?
> 
> 
> Regards
> Matthias
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: avg apply latency went up after update from octopus to pacific

[Konstantin Shalygin]

Hi,


Maybe [1] ?


[1] https://tracker.ceph.com/issues/58530
k

> On 22 Mar 2023, at 16:20, Boris Behrens  wrote:
> 
> Are there any other ides?
> 

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: s3 compatible interface

[Konstantin Shalygin]

Hi,

You can try [1] geesefs project, the presentation for this code is here [2]


[1] https://github.com/yandex-cloud/geesefs
[2] 
https://yourcmc-ru.translate.goog/geesefs-2022/highload.html?_x_tr_sl=ru&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp
k

> On 28 Feb 2023, at 22:31, Marc  wrote:
> 
> Anyone know of a s3 compatible interface that I can just run, and 
> reads/writes files from a local file system and not from object storage?

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph noout vs ceph norebalance, which is better for minor maintenance

[Konstantin Shalygin]

> On 17 Feb 2023, at 23:20, Anthony D'Atri  wrote:
> 
> 
> 
>> * if rebalance will starts due EDAC or SFP degradation, is faster to fix the 
>> issue via DC engineers and put node back to work
> 
> A judicious mon_osd_down_out_subtree_limit setting can also do this by not 
> rebalancing when an entire node is detected down. 

Yes. But in this case when single disk dead, it's may be not actually dead, the 
examples:

* disk just stuck - reboot or/and physical inject_insert return in to live
* disk read errors - such errors lead to OSD down, but after OSD restart is 
just works normal (Pending Sectors -> Reallocates)

The fill of single 16TB OSD may be a 7-10 days. And it's may be fixed with 
10-20 minutes with duty engineer

> 
>> * noout prevents unwanted OSD's fills and the run out of space => outage of 
>> services
> 
> Do you run your clusters very full?

We provide public services. This means client can rent 1000 disks x 1000GB via 
one terraform command, at 02:00 Saturday night. Just physically impossible to 
add nodes at this case. Any movement without upmap is highly undesirable



k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph noout vs ceph norebalance, which is better for minor maintenance

[Konstantin Shalygin]

> On 17 Feb 2023, at 23:20, Anthony D'Atri  wrote:
> 
> 
> 
>> * if rebalance will starts due EDAC or SFP degradation, is faster to fix the 
>> issue via DC engineers and put node back to work
> 
> A judicious mon_osd_down_out_subtree_limit setting can also do this by not 
> rebalancing when an entire node is detected down. 

Yes. But in this case when single disk dead, it's may be not actually dead, the 
examples:

* disk just stuck - reboot or/and physical inject_insert return in to live
* disk read errors - such errors lead to OSD down, but after OSD restart is 
just works normal (Pending Sectors -> Reallocates)

The fill of single 16TB OSD may be a 7-10 days. And it's may be fixed with 
10-20 minutes with duty engineer

> 
>> * noout prevents unwanted OSD's fills and the run out of space => outage of 
>> services
> 
> Do you run your clusters very full?

We provide public services. This means client can rent 1000 disks x 1000GB via 
one terraform command, at 02:00 Saturday night. Just physically impossible to 
add nodes at this case. Any movement without upmap is highly undesirable



k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph noout vs ceph norebalance, which is better for minor maintenance

[Konstantin Shalygin]

Hi Will,

All our clusters with noout flag by default, since cluster birth. The reasons:

* if rebalance will starts due EDAC or SFP degradation, is faster to fix the 
issue via DC engineers and put node back to work

* noout prevents unwanted OSD's fills and the run out of space => outage of 
services

* OSD down (broken disk) Prometeus alert will be resolved without noout - after 
OSD will be marked out. Because UP state of OSD in metrics world is expression 
of (in + up). We need "fire alert" for humans, for disk replacement 


Also hope this helps!

k
Sent from my iPhone

> On 16 Feb 2023, at 06:30, William Konitzer  wrote:
> Hi Dan,
> 
> I appreciate the quick response. In that case, would something like this be 
> better, or is it overkill!?
> 
> 1. ceph osd add-noout osd.x #mark out for recovery operations
> 2. ceph osd add-noin osd.x #prevent rebalancing onto the OSD
> 3. kubectl -n rook-ceph scale deployment rook-ceph-osd--* --replicas=0 
> #disable OSD
> 4. ceph osd down osd.x #prevent it from data placement and recovery operations
> 5. Upgrade the firmware on OSD
> 6. ceph osd up osd.x
> 7. kubectl -n rook-ceph scale deployment rook-ceph-osd--* --replicas=1
> 8. ceph osd rm-noin osd.x
> 9. ceph osd rm-noout osd.x
> 
> Thanks,
> Will
> 
> 
>> On Feb 15, 2023, at 5:05 PM, Dan van der Ster  wrote:
>> 
>> Sorry -- Let me rewrite that second paragraph without overloading the
>> term "rebalancing", which I recognize is confusing.
>> 
>> ...
>> 
>> In your case, where you want to perform a quick firmware update on the
>> drive, you should just use noout.
>> 
>> Without noout, the OSD will be marked out after 5 minutes and objects
>> will be re-replicated to other OSDs -- those degraded PGs will move to
>> "backfilling" state and copy the objects on new OSDs.
>> 
>> With noout, the cluster won't start backfilling/recovering, but don't
>> worry -- this won't block IO. What happens is the disk that is having
>> its firmware upgraded will be marked "down", and IO will be accepted
>> and logged by its peers, so that when the disk is back "up" it can
>> replay ("recover") those writes to catch up.
>> 
>> 
>> The norebalance flag only impacts data movement for PGs that are not
>> degraded -- no OSDs are down. This can be useful to pause backfilling
>> e.g. when you are adding or removing hosts to a cluster.
>> 
>> -- dan
>> 
>> On Wed, Feb 15, 2023 at 2:58 PM Dan van der Ster  wrote:
>>> Hi Will,
>>> There are some misconceptions in your mail.
>>> 1. "noout" is a flag used to prevent the down -> out transition after
>>> an osd is down for several minutes. (Default 5 minutes).
>>> 2. "norebalance" is a flag used to prevent objects from being
>>> backfilling to a different OSD *if the PG is not degraded*.
>>> In your case, where you want to perform a quick firmware update on the
>>> drive, you should just use noout.
>>> Without noout, the OSD will be marked out after 5 minutes and data
>>> will start rebalancing to other OSDs.
>>> With noout, the cluster won't start rebalancing. But this won't block
>>> IO -- the disk being repaired will be "down" and IO will be accepted
>>> and logged by it's peers, so that when the disk is back "up" it can
>>> replay those writes to catch up.
>>> Hope that helps!
>>> Dan
>>> On Wed, Feb 15, 2023 at 1:12 PM  wrote:
 Hi,
 We have a discussion going on about which is the correct flag to use for 
 some maintenance on an OSD, should it be "noout" or "norebalance"? This 
 was sparked because we need to take an OSD out of service for a short 
 while to upgrade the firmware.
 One school of thought is:
 - "ceph norebalance" prevents automatic rebalancing of data between OSDs, 
 which Ceph does to ensure all OSDs have roughly the same amount of data.
 - "ceph noout" on the other hand prevents OSDs from being marked as 
 out-of-service during maintenance, which helps maintain cluster 
 performance and availability.
 - Additionally, if another OSD fails while the "norebalance" flag is set, 
 the data redundancy and fault tolerance of the Ceph cluster may be 
 compromised.
 - So if we're going to maintain the performance and reliability we need to 
 set the "ceph noout" flag to prevent the OSD from being marked as OOS 
 during maintenance and allow the automatic data redistribution feature of 
 Ceph to work as intended.
 The other opinion is:
 - With the noout flag set, Ceph clients are forced to think that OSD 
 exists and is accessible - so they continue sending requests to such OSD. 
 The OSD also remains in the crush map without any signs that it is 
 actually out. If an additional OSD fails in the cluster with the noout 
 flag set, Ceph is forced to continue thinking that this new failed OSD is 
 OK. It leads to stalled or delayed response from the OSD side to clients.
 - Norebalance instead takes into account the in/out OSD status, but 
 prevents data rebalance. Clients are 

[ceph-users] Re: iDRAC 9 version 6.10 shows 0% for write endurance on non-dell drives, work around?

[Konstantin Shalygin]


Hi,

You can use smartctl_exporter [1] for all your media, not only the SSD


k
[1] https://github.com/prometheus-community/smartctl_exporter

Sent from my iPhone

> On 14 Feb 2023, at 23:01, Drew Weaver  wrote:
> Hello,
> 
> After upgrading a lot of iDRAC9 modules to version 6.10 in servers that are 
> involved in a Ceph cluster we noticed that the iDRAC9 shows the write 
> endurance as 0% on any non-certified disk.
> 
> OMSA still shows the correct remaining write endurance but I am assuming that 
> they are working feverishly to eliminate that too.
> 
> I opened a support ticket with Dell once this was brought to my attention and 
> they basically told me that I was lucky that it ever worked at all, which I 
> thought was an odd response given that the iDRAC enterprise licenses cost 
> several hundred dollars each.
> 
> I know that the old Intel Datacenter Tool used to be able to reach through a 
> MegaRAID controller and read the remaining write endurance but that tool is 
> essentially defunct now.
> 
> What are you folks using to monitor your write endurance on your SSDs that 
> you couldn't buy from Dell because they had a 16 week lead time while the MFG 
> could deliver the drives in 3 days?
> 
> Thanks,
> -Drew
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Quincy On Rocky 8.x - Upgrade To Rocky 9.1

[Konstantin Shalygin]

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Quincy On Rocky 8.x - Upgrade To Rocky 9.1

[Konstantin Shalygin]

Hi,

Seems packages el9_quincy are available [1]
You can try


k
[1] https://download.ceph.com/rpm-quincy/el9/x86_64/

> On 10 Feb 2023, at 13:23, duluxoz  wrote:
> 
> Sorry if this was mentioned previously (I obviously missed it if it was) but 
> can we upgrade a Ceph Quincy Host/Cluster from Rocky Linux (RHEL) v8.6/8.7 to 
> v9.1 (yet), and if so, what is / where can I find the procedure to do this - 
> ie is there anything "special" that needs to be done because of Ceph, or can 
> we just do a "simple" v8.x +> v9.1 upgrade?

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: January Ceph Science Virtual User Group

[Konstantin Shalygin]

Hi Mark,

Why you need to upgrade every year? 
Yes, scalable - you can add new racks without new version or new distro - that 
how big clusters are live


k

Sent from my iPhone

> On 2 Feb 2023, at 19:09, Marc  wrote:
> 
> 
>> 
>> 
>> https://www.youtube.com/playlist?list=PLrBUGiINAakM3d4bw6Rb7EZUcLd98iaWG
>> 
> 
> Interesting to hear about the container environment not able to scale to 
> well. 
> 
> And listening to this I started to wonder also about the current release 
> cycle. I can remember the discussion about this, but to me it looks like 
> almost all big clusters can't keep up with upgrading every year. I am not 
> even looking forward to upgrading my tiny cluster.
> 
> Yet even on the ceph.com home page, one of the key features advertised is 
> scalable .
> 
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm upgrade 16.2.10 to 16.2.11: osds crash and get stuck restarting

[Konstantin Shalygin]

Hi Zakhar,

> On 26 Jan 2023, at 08:33, Zakhar Kirpichenko  wrote:
> 
> Jan 25 23:07:53 ceph01 bash[2553123]:
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc
>  :
> 604: ceph_abort_msg("Unexpected IO error. This may suggest HW issue. Please
> check your dmesg!")

You can check your kmesg for a messages via `journalctl -kl --since=yesterday` 
to see what actually happened with this device


k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Status of Quincy 17.2.5 ?

[Konstantin Shalygin]

May be Mike can organize this release flow... 
CC'ed Mike Perez, I think team need some manager observability (a little)


k

> On 25 Jan 2023, at 16:26, Christian Rohmann  
> wrote:
> 
> Hey everyone,
> 
> 
> On 20/10/2022 10:12, Christian Rohmann wrote:
>> 1) May I bring up again my remarks about the timing:
>> 
>> On 19/10/2022 11:46, Christian Rohmann wrote:
>> 
>>> I believe the upload of a new release to the repo prior to the announcement 
>>> happens quite regularly - it might just be due to the technical process of 
>>> releasing.
>>> But I agree it would be nice to have a more "bit flip" approach to new 
>>> releases in the repo and not have the packages appear as updates prior to 
>>> the announcement and final release and update notes.
>> By my observations sometimes there are packages available on the download 
>> servers via the "last stable" folders such as 
>> https://download.ceph.com/debian-quincy/ quite some time before the 
>> announcement of a release is out.
>> I know it's hard to time this right with mirrors requiring some time to sync 
>> files, but would be nice to not see the packages or have people install them 
>> before there are the release notes and potential pointers to changes out. 
> 
> Todays 16.2.11 release shows the exact issue I described above 
> 
> 1) 16.2.11 packages are already available via e.g. 
> https://download.ceph.com/debian-pacific
> 2) release notes not yet merged: (https://github.com/ceph/ceph/pull/49839), 
> thus https://ceph.io/en/news/blog/2022/v16-2-11-pacific-released/ show a 404 
> :-)
> 3) No announcement like 
> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/QOCU563UD3D3ZTB5C5BJT5WRSJL5CVSD/
>  to the ML yet.
> 
> 
> Regards
> 
> 
> Christian
> 
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph cluster iops low

[Konstantin Shalygin]

Hi,

You SSD is a "desktop" SSD, not a "enterprise" SSD, see [1]
This mostly was't suitable for Ceph


[1] https://yourcmc.ru/wiki/Ceph_performance#CAPACITORS.21

k

> On 25 Jan 2023, at 05:35, peter...@raksmart.com wrote:
> 
> Hi Mark,
> Thanks for your response, it is help!
> Our Ceph cluster use Samsung SSD 870 EVO all backed with NVME drive. 12 SSD 
> drives to 2 NVMe drives per storage node. Each 4TB SSD backed 283G NVMe lvm 
> partition as DB. 
> Now cluster throughput only 300M write, and around 5K IOPS.  I could see NVMe 
> drive utilization over 95% show on ‘iostat’ command. Will NVMe drive be a 
> bottle neck quickly if we have large of IO in cluster?
> I have read the top article about OSD bundle with CPU cores. However I can 
> only find script called pincpu on the github to automate process to allocate 
> CPU core with OSDs. It seems not work for me. Do you have any tool or 
> official instruction that can guide me to test it?

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Mount ceph using FQDN

[Konstantin Shalygin]

Hi,

Do you think kernel should care about DNS resolution?


k

> On 24 Jan 2023, at 19:07, kushagra.gu...@hsc.com wrote:
> 
> Hi team,
> 
> We have a ceph cluster with 3 storage nodes:
> 1. storagenode1 - abcd:abcd:abcd::21
> 2. storagenode2 - abcd:abcd:abcd::22
> 3. storagenode3 - abcd:abcd:abcd::23
> 
> We have a dns server with ip abcd:abcd:abcd::31 which resolves the above ip's 
> with a single hostname.
> The resolution is as follows:
> ```
> $TTL 1D
> @   IN SOA  storage.com   root (
>6   ; serial
>1D  ; refresh
>1H  ; retry
>1W  ; expire
>3H ); minimum
> 
>  INNS  master
> master   INA10.0.1.31
> storagenode  IN    abcd:abcd:abcd::21
> storagenode  IN    abcd:abcd:abcd::22
> storagenode  IN    abcd:abcd:abcd::23
> ```
> 
> We want to mount the ceph storage on a node using this hostname.
> For this we are using the command:
> ```
> mount -t ceph [storagenode.storage.com]:6789:/  /backup -o 
> name=admin,secret=AQCM+8hjqzuZEhAAcuQc+onNKReq7MV+ykFirg==
> ```
> 
> We are getting the following logs in /var/log/messages:
> ```
> Jan 24 17:23:17 localhost kernel: libceph: resolve 'storagenode.storage.com' 
> (ret=-3): failed
> Jan 24 17:23:17 localhost kernel: libceph: parse_ips bad ip 
> 'storagenode.storage.com:6789'
> ```
> 
> We also tried mounting ceph storage by removing the dns server and resolving 
> the ip as follows:
> ```
> abcd:abcd:abcd::21 storagenode1
> ```
> 
> But we are getting similar results.
> 
> Also kindly note that we are able to perform the mount operation if we use 
> ips instead of domain name.
> 
> Could you please help us out with how we can mount ceph using FQDN.
> 
> Kindly let me know if any other imformation is required.
> 
> My ceph.conf configuration is as follows:
> ```
> [global]
> ms bind ipv6 = true
> ms bind ipv4 = false
> mon initial members = storagenode1,storagenode2,storagenode3
> osd pool default crush rule = -1
> fsid = 7969b8a3-1df7-4eae-8ccf-2e5794de87fe
> mon host = 
> [v2:[abcd:abcd:abcd::21]:3300,v1:[abcd:abcd:abcd::21]:6789],[v2:[abcd:abcd:abcd::22]:3300,v1:[abcd:abcd:abcd::22]:6789],[v2:[abcd:abcd:abcd::23]:3300,v1:[abcd:abcd:abcd::23]:6789]
> public network = abcd:abcd:abcd::/64
> cluster network = eff0:eff0:eff0::/64
> 
> [osd]
> osd memory target = 4294967296
> 
> [client.rgw.storagenode1.rgw0]
> host = storagenode1
> keyring = /var/lib/ceph/radosgw/ceph-rgw.storagenode1.rgw0/keyring
> log file = /var/log/ceph/ceph-rgw-storagenode1.rgw0.log
> rgw frontends = beast endpoint=[abcd:abcd:abcd::21]:8080
> rgw thread pool size = 512
> ```
> 
> Thanks and Regards
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Current min_alloc_size of OSD?

[Konstantin Shalygin]

Hi,

> On 12 Jan 2023, at 04:35, Robert Sander  wrote:
> 
> How can I get the current min_allloc_size of OSDs that were created with 
> older Ceph versions? Is there a command that shows this info from the on disk 
> format of a bluestore OSD?

You can see this via kvstore-tool:


ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-0/ get S min_alloc_size

4096:

  00 10 00 00 00 00 00 00   ||
0008

65536:

  00 00 01 00 00 00 00 00   ||
0008


Cheers,
k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [ERR] OSD_SCRUB_ERRORS: 2 scrub errors

[Konstantin Shalygin]

Hi,

> On 10 Jan 2023, at 07:10, David Orman  wrote:
> 
> We ship all of this to our centralized monitoring system (and a lot more) and 
> have dashboards/proactive monitoring/alerting with 100PiB+ of Ceph. If you're 
> running Ceph in production, I believe host-level monitoring is critical, 
> above and beyond Ceph level. Things like inlet/outlet temperature, hardware 
> state of various components, and various other details are probably best 
> served by monitoring external to Ceph itself.

I agree with David's suggestions

> 
> I did a quick glance and didn't see this data (OSD errors re: reads/writes) 
> exposed in the Pacific version of Ceph's Prometheus-style exporter, but I may 
> have overlooked it. This would be nice to have, as well, if it does not exist.
> 
> We collect drive counters at the host level, and alert at levels prior to 
> general impact. Even a failing drive can cause latency spikes which are 
> frustrating, before it starts returning errors (correctable errors) - the OSD 
> will not see these other than longer latency on operations. Seeing a change 
> in the smart counters either at a high rate or above thresholds you define is 
> most certainly something I would suggest ensuring is covered in whatever 
> host-level monitoring you're already performing for production usage.

Seems to me that there is no need to reinvent the wheel and create even more 
GIL problems for ceph-mgr. In previous year was released production-ready 
exporter for smartctl data, with NVMe support [1]
Golang, CI & tested in production with Ceph - ready to go 


[1] https://github.com/prometheus-community/smartctl_exporter
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: max pool size (amount of data/number of OSDs)

[Konstantin Shalygin]

Hi Chris,

The actually limits are not software. Usually Ceph teams on Cloud Providers or 
Universities running out at physical resources at first: racks, racks power or 
network (ports, EOL switches that can't be upgraded) or hardware lifetime 
(There is no point in buying old hardware, and the new one is too new to 
interfere with the old one.  At the same time, replacing everything at once is 
very expensive (millions of dollars [depends on the region where the equipment 
is purchased and where it will be operated]))


k
Sent from my iPhone

> On 30 Dec 2022, at 19:52, Christopher Durham  wrote:
> 
> 
> Hi,
> Is there any information on this issue? Max number of OSDs per pool, or 
> maxpool size (data) as opposed to cluster size? Thanks!
> -Chris
> 
> 
> -Original Message-
> From: Christopher Durham 
> To: ceph-users@ceph.io 
> Sent: Thu, Dec 15, 2022 5:36 pm
> Subject: max pool size (amount of data/number of OSDs)
> 
> 
> Hi,
> There are various articles, case studies, etc about large ceph clusters, 
> storing 10s of PiB,with CERN being the largest cluster as far as I know.
> Is there a largest pool capacity limit?  In other words, while you may have a 
> 30PiB cluster,is there a limit or recommendation as to max pool capacity. For 
> example, in the 30PiB example,is there a limit or recommendation that says do 
> not have a pool capacity of higher than 5iB, for 6pools in that cluster at a 
> ttotal of 30PiB?
> 
> I know this would be contingent upon a variety of things, including, but not 
> limited to network throughput, individual serversize (disk size and number, 
> memory, compute). I am specifically talking about s3./rgw storage.
> 
> But is there a technical limit, or just a tested size, of a pool? Should I 
> createdifferent pools when a given pool would otherwise reach a size capacity 
> of Xor have N osds or PGs in it, when considering adding additional osds?
> Thanks for any info
> -Chris
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: radosgw not working after upgrade to Quincy

[Konstantin Shalygin]

Hi,

Just try to read your logs:

> 2022-12-29T02:07:38.953+ 7f5df868ccc0 0 WARNING: skipping unknown 
> framework: civetweb 

You try to use the `civetweb`, it was absent in quincy release. You need to 
update your configs and use `beast` instead


k

> On 29 Dec 2022, at 09:20, Andrei Mikhailovsky  wrote:
> 
> Please let me know how to fix the problem? 

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Does Replica Count Affect Tell Bench Result or Not?

[Konstantin Shalygin]

Hi,

The cache was gone, optimize is proceed. This is not enterprise device, you 
should never use it with Ceph 


k
Sent from my iPhone

> On 27 Dec 2022, at 16:41, hosseinz8...@yahoo.com wrote:
> 
>  Thanks AnthonyI have a cluster with QLC SSD disks (Samsung QVO 860). The 
> cluster works for 2 year. Now all OSDs return 12 iops when running tell bench 
> which is very slow. But I Buy new QVO disks yesterday, and I added this new 
> disk to cluster. For the first 1 hour, I got 100 iops from this new OSD. But 
> after 1 Hour, this new disk (OSD) returns to iops 12 again as the same as 
> other OLD OSDs.I can not imagine what happening?!!
>On Tuesday, December 27, 2022 at 12:18:07 AM GMT+3:30, Anthony D'Atri 
>  wrote:  
> 
> My understanding is that when you ask an OSD to bench (via the admin socket), 
> only that OSD executes, there is no replication.  Replication is a function 
> of PGs.
> 
> Thus, this is a narrowly-focused tool with both unique advantages and 
> disadvantages.
> 
> 
> 
>> On Dec 26, 2022, at 12:47 PM, hosseinz8...@yahoo.com wrote:
>> 
>> Hi experts,I want to know, when I execute ceph tell osd.x bench command, is 
>> replica 3 considered in the bench or not? I mean, for example in case of 
>> replica 3, when I executing tell bench command, replica 1 of bench data 
>> write to osd.x, replica 2 write to osd.y and replica 3 write to osd.z? If 
>> this is true, it means that I can not get benchmark of only one of my OSD in 
>> the cluster because the IOPS and throughput of 2 other for example slow OSDs 
>> will affect the result of tell bench command for my target OSD.Is that true?
>> Thanks in advance.
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Empty /var/lib/ceph/osd/ceph-$osd after reboot

[Konstantin Shalygin]

Hi,

ceph-volume lvm activate --all


k

> On 21 Dec 2022, at 13:53, Isaiah Tang Yue Shun  wrote:
> 
> From what I understand, after creating an OSD using  "ceph-volume lvm 
> create", we will do a "ceph-volume lvm activate" so that the systemd is 
> created.
> 
> However, I found that after rebooting a host, some OSDs in the host will have 
> empty /var/lib/ceph/osd/ceph-$osd
> And I am not able to recover from there. Am I missing any steps?
> 
> I am running Oracle Linux 8.7, pacific release, ol8.
> 
> Thanks in advance.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io