[ceph-users] Re: Ceph client cluster compatibility

Hi, Yes, like it always do k Sent from my iPhone > On 2 May 2024, at 07:09, Nima AbolhassanBeigi > wrote: > > We are trying to upgrade our OS version from ubuntu 18.04 to ubuntu 22.04. > Our ceph cluster version is 16.2.13 (pacific). > > The problem is that the ubuntu packages for the ceph

[ceph-users] Re: Ceph image delete error - NetHandler create_socket couldnt create socket

Hi, > On 19 Apr 2024, at 10:39, Pardhiv Karri wrote: > > Thank you for the reply. I tried setting ulimit to 32768 when I saw 25726 > number in lsof output and then after 2 disks deletion again it got an error > and checked lsof and which is above 35000. I'm not sure how to handle it. > I

[ceph-users] Re: Ceph image delete error - NetHandler create_socket couldnt create socket

Hi, Your shell seems reached the default file discriptors limit (1024 mostly) and your cluster maybe more than 1000 OSD Try to set command `ulimit -n 10240` before rbd rm task k Sent from my iPhone > On 18 Apr 2024, at 23:50, Pardhiv Karri wrote: > > Hi, > > Trying to delete images in a

[ceph-users] Re: Client kernel crashes on cephfs access

Hi Xiubo, Seems patch already landed to kernel 6.8.7, thanks! k Sent from my iPhone > On 18 Apr 2024, at 05:31, Xiubo Li wrote: > > Hi Konstantin, > > We have fixed it, please see > https://patchwork.kernel.org/project/ceph-devel/list/?series=842682=both. > > - Xiubo

[ceph-users] Re: Client kernel crashes on cephfs access

Hi, > On 9 Apr 2024, at 04:07, Xiubo Li wrote: > > Thanks for reporting this, I generated one patch to fix it. Will send it out > after testing is done. Trace from our users, but from mainline kernel. Look like as trace above kernel: [ cut here ] kernel: list_add

[ceph-users] Re: Impact of large PG splits

> On 10 Apr 2024, at 01:00, Eugen Block wrote: > > I appreciate your message, it really sounds tough (9 months, really?!). But > thanks for the reassurance :-) Yes, the total "make this project great again" tooks 16 month, I think. This my work First problem after 1M objects in PG was a

[ceph-users] Re: Impact of large PG splits

Hi Eugene! I have a case, where PG with millions of objects, like this ``` root@host# ./show_osd_pool_pg_usage.sh | less | head id used_mbytes used_objects omap_used_mbytes omap_used_keys -- --- -- 17.c91

[ceph-users] Re: Ceph object gateway metrics

Hi, You can use the [2] exporter to achieve usage stats per user and per bucket, including quotas usage k Sent from my iPhone > On 26 Mar 2024, at 01:38, Kushagr Gupta wrote: > > 2. https://github.com/blemmenes/radosgw_usage_exporter ___

[ceph-users] Re: Laptop Losing Connectivity To CephFS On Sleep/Hibernation

Hi, Yes, this is generic solution for end users mounts - samba gateway k Sent from my iPhone > On 23 Mar 2024, at 12:10, duluxoz wrote: > > Hi Alex, and thanks for getting back to me so quickly (I really appreciate > it), > > So from what you said it looks like we've got the wrong

[ceph-users] Re: RGW - tracking new bucket creation and bucket usage

Hi, > On 15 Mar 2024, at 01:07, Ondřej Kukla wrote: > > Hello I’m looking for suggestions how to track bucket creation over s3 api > and bucket usage (num of objects and size) of all buckets in time. > > In our RGW setup, we have a custom client panel, where like 85% percent of > buckets are

[ceph-users] Re: ceph metrics units

Hi, > On 14 Mar 2024, at 19:29, Denis Polom wrote: > > so metric itself is miliseconds and after division on _count it's in seconds? > > This is two metrics for long running averages [1], the query that produces "seconds" unit looks like this (irate(ceph_osd_op_r_latency_sum[1m]) /

[ceph-users] Re: ceph metrics units

Hi, > On 14 Mar 2024, at 16:44, Denis Polom wrote: > > do you know if there is some table of Ceph metrics and units that should be > used for them? > > I currently struggling with > > ceph_osd_op_r_latency_sum > > ceph_osd_op_w_latency_sum > > if they are in ms or seconds? > > Any idea

[ceph-users] Re: Telemetry endpoint down?

Hi Greg Seems is up now, last report uploaded successfully Thanks, k Sent from my iPhone > On 11 Mar 2024, at 18:57, Gregory Farnum wrote: > > We had a lab outage Thursday and it looks like this service wasn’t > restarted after that occurred. Fixed now and we’ll look at how to prevent > that

[ceph-users] Telemetry endpoint down?

Hi, seems telemetry endpoint is down for a some days? We have connection errors from multiple places 1:ERROR Mar 10 00:46:10.653 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 2:ERROR Mar 10 01:48:20.061 [564383]: opensock: Could not establish a

[ceph-users] Re: Running dedicated RGWs for async tasks

Hi, Yes. You need to turn off gc, lc threads in config for your current (client side) RGW's. Then setup your 'async tasks' RGW without client traffic. No special configuration needed, only if I wanna tune gc, lc settings k Sent from my iPhone > On 7 Mar 2024, at 13:09, Marc Singer wrote: >

[ceph-users] Re: ceph Quincy to Reef non cephadm upgrade

Hi, Yes, you upgrade ceph-common package, then restart your mons k Sent from my iPhone > On 6 Mar 2024, at 21:55, sarda.r...@gmail.com wrote: > > My question is - does this mean I need to upgrade all ceph packages (ceph, > ceph-common) and restart only monitor daemon first?

[ceph-users] Re: Monitoring Ceph Bucket and overall ceph cluster remaining space

Hi, Don't aware about what is SW, but if this software works with Prometheus metrics format - why not. Anyway the exporters are open source, you can modify the existing code for your environment k Sent from my iPhone > On 6 Mar 2024, at 07:58, Michael Worsham wrote: > > This looks

[ceph-users] Re: Monitoring Ceph Bucket and overall ceph cluster remaining space

Hi, For RGW usage statistics you can use radosgw_usage_exporter [1] k [1] https://github.com/blemmenes/radosgw_usage_exporter Sent from my iPhone > On 6 Mar 2024, at 00:21, Michael Worsham wrote: > Is there an easy way to poll the ceph cluster buckets in a way to see how > much space is

[ceph-users] Re: Size return by df

Hi, Yes you can, this controlled by option client quota df = false k Sent from my iPhone > On Feb 22, 2024, at 11:17, Albert Shih wrote: > > Is they are any way to keep the first answer ? ___ ceph-users mailing list -- ceph-users@ceph.io To

[ceph-users] Re: pacific 16.2.15 QE validation status

> > On Feb 7, 2024, at 16:59, Zakhar Kirpichenko wrote: > > Indeed, it looks like it's been recently reopened. Thanks for this! Hi, It was merged yesterday Thanks for the right noise, k ___ ceph-users mailing list -- ceph-users@ceph.io To

[ceph-users] Re: Ceph 16.2.14: ceph-mgr getting oom-killed

Hi, The backport to pacific was rejected [1], you may switch to reef, when [2] merged and released [1] https://github.com/ceph/ceph/pull/55109 [2] https://github.com/ceph/ceph/pull/55110 k Sent from my iPhone > On Jan 25, 2024, at 04:12, changzhi tan <544463...@qq.com> wrote: > > Is there

[ceph-users] Re: Ceph Nautilous 14.2.22 slow OSD memory leak?

Hi, > On Jan 12, 2024, at 12:01, Frédéric Nass > wrote: > > Hard to tell for sure since this bug hit different major versions of the > kernel, at least RHEL's from what I know. In what RH kernel release this issue was fixed? Thanks, k ___

[ceph-users] Re: MDS recovery with existing pools

Good to hear that, Eugen! CC'ed Zac for a your docs mention k > On Dec 11, 2023, at 23:28, Eugen Block wrote: > > Update: apparently, we did it! > We walked through the disaster recovery steps where one of the steps was to > reset the journal. I was under the impression that the specified

[ceph-users] Re: osdmaptool target & deviation calculation

Hi, This deviation is very soft. If u wanna do real upmaps you should use deviation 1 k Sent from my iPhone > On Nov 27, 2023, at 21:39, Robert Hish wrote: > > The result is many many OSDs with a deviation well above the > upmap_max_deviation which is at default: 5

[ceph-users] Re: CLT Meeting minutes 2023-11-23

Hi, > On Nov 23, 2023, at 16:10, Nizamudeen A wrote: > > RCs for reef, quincy and pacific > for next week when there is more time to discuss Just little noise: pacific is ready? 16.2.15 should be last release (at least that was the last plan), but [1] still not merged. Why now ticket is

[ceph-users] Re: Bug fixes in 17.2.7

Hi, > On Nov 20, 2023, at 19:24, Tobias Kulschewski > wrote: > > do you have a rough estimate of when this will happen? > > Not at this year I think. For now precedence for a 18.2.1 and last release of Pacific But you can request shaman build, and clone repo for your local usage k

[ceph-users] Re: Bug fixes in 17.2.7

Hi Tobias, This was not meged to Quincy yet [1] k [1] https://tracker.ceph.com/issues/59730 Sent from my iPhone > On Nov 20, 2023, at 17:50, Tobias Kulschewski > wrote: > > Just wanted to ask, if the bug with the multipart upload [1] has been fixed > in 17.2.7?

[ceph-users] Re: HDD cache

Hi Peter, > On Nov 8, 2023, at 20:32, Peter wrote: > > Anyone experienced this can advise? You can try: * check for current cache status smartctl -x /dev/sda | grep "Write cache" * turn off write cache smartctl -s wcache-sct,off,p /dev/sda * check again smartctl -x /dev/sda | grep "Write

[ceph-users] Re: Problem: Upgrading CEPH Pacific to Quincy resulted in CEPH Storage pool to stop functioning.

Hi, You need revert your packages from Quincy to Pacific. `dnf downgrade ceph-mon` command should help with this k Sent from my iPhone > On Oct 11, 2023, at 03:22, Waywatcher wrote: > > I am unable to get any of the current monitors to run. They all fail to start

[ceph-users] Re: Problem: Upgrading CEPH Pacific to Quincy resulted in CEPH Storage pool to stop functioning.

Hi, For this upgrade you need at least some mon's up, then you can redeploy your pacific mon's to rocksdb k Sent from my iPhone > On Oct 10, 2023, at 02:01, Waywatcher wrote: > > I upgraded my CEPH cluster without properly following the mon upgrade so > they were no longer on leveldb. > >

[ceph-users] Re: replacing storage server host (not drives)

Hi, The procedure is simple: get another host and put current disk to new host. Setup boot and network's and back to business k Sent from my iPhone > On Sep 26, 2023, at 17:38, Wyll Ingersoll > wrote: > > What is the recommended procedure for replacing the host itself without > destroying

[ceph-users] Re: cephfs mount 'stalls'

Hi, William, this is fuse client, not the kernel Mark, you can use kernel client. Stock c7 or install, for example, kernel-ml from ELrepo [1], and use the latest krbd version [1] https://elrepo.org/tiki/HomePage#elrepo-kernel k Sent from my iPhone > On 17 Sep 2023, at 22:10, William Edwards

[ceph-users] Re: Rocksdb compaction and OSD timeout

Hi Igor, > On 12 Sep 2023, at 15:28, Igor Fedotov wrote: > > Default hybrid allocator (as well as AVL one it's based on) could take > dramatically long time to allocate pretty large (hundreds of MBs) 64K-aligned > chunks for BlueFS. At the original cluster it was exposed as 20-30 sec OSD >

[ceph-users] Re: Rocksdb compaction and OSD timeout

This cluster use the default settings or something for Bluestore was changed? You can check this via `ceph config diff` As Mark said, it will be nice to have a tracker, if this really release problem Thanks, k Sent from my iPhone > On 7 Sep 2023, at 20:22, J-P Methot wrote: > > We went from

[ceph-users] Re: Rocksdb compaction and OSD timeout

Hi, > On 7 Sep 2023, at 18:21, J-P Methot wrote: > > Since my post, we've been speaking with a member of the Ceph dev team. He > did, at first, believe it was an issue linked to the common performance > degradation after huge deletes operation. So we did do offline compactions on > all our

[ceph-users] Re: Rocksdb compaction and OSD timeout

Hi, > On 7 Sep 2023, at 10:05, J-P Methot wrote: > > We're running latest Pacific on our production cluster and we've been seeing > the dreaded 'OSD::osd_op_tp thread 0x7f346aa64700' had timed out after > 15.00954s' error. We have reasons to believe this happens each time the > RocksDB

[ceph-users] Re: Status of diskprediction MGR module?

Hi, > On 28 Aug 2023, at 12:45, Robert Sander wrote: > > Several years ago the diskprediction module was added to the MGR collecting > SMART data from the OSDs. > > There were local and cloud modes available claiming different accuracies. Now > only the local mode remains. > > What is the

[ceph-users] Re: User + Dev Monthly Meeting Minutes 2023-08-24

On 24 Aug 2023, at 18:51, Laura Flores wrote: > > Need pacific backport for https://tracker.ceph.com/issues/59478 > > - Laura will check on this, although a Pacific backport is unlikely due > to incompatibilities from the scrub backend refactoring. Laura, this fix "for malformed fix" of

[ceph-users] Re: [quincy] Migrating ceph cluster to new network, bind OSDs to multple public_nework

Hi, This how OSD's woks. For change the network subnet you need to setup reachability of both: old and new network, until end of migration k Sent from my iPhone > On 22 Aug 2023, at 10:43, Boris Behrens wrote: > > The OSDs are still only bound to one IP address.

[ceph-users] Re: Check allocated RGW bucket/object size after enabling Bluestore compression

Hi, > On 17 Aug 2023, at 18:21, yosr.kchao...@gmail.com wrote: > > Thanks for your reply. By the Bluestore compression I mean the compression > enabled on the pool level. It is also called inline compression. >

[ceph-users] Re: Check allocated RGW bucket/object size after enabling Bluestore compression

Hi, What you mean, Bluestore compression? The rgw compression is a rados compression, not the compress by rgw itself. You can setup different storage classes and upload to same pool uncompressed, or compressed objects The compression ratio you can determine with exporter [1] [1]

[ceph-users] Re: Can't join new mon - lossy channel, failing

> On 16 Aug 2023, at 13:23, Josef Johansson wrote: > > I'm running ceph version 15.2.16 (a6b69e817d6c9e6f02d0a7ac3043ba9cdbda1bdf) > octopus (stable), that would mean I am not running the fix. > > Glad to know that an upgrade will solve the issue! I'm not 100% sure that this tracker,

[ceph-users] Re: Can't join new mon - lossy channel, failing

Hi, > On 16 Aug 2023, at 11:30, Josef Johansson wrote: > > Let's do some serious necromancy here. > > I just had this exact problem. Turns out that after rebooting all nodes (one > at the time of course), the monitor could join perfectly. > > Why? You tell me. We did not see any traces of

[ceph-users] Re: v18.2.0 Reef released

Hi, Thanks for the release! Please, upgrade the OS Platform docs, currently there lack of Reef in ABC tests: https://tracker.ceph.com/issues/62354 Thanks, k > On 7 Aug 2023, at 21:37, Yuri Weinstein wrote: > > We're very happy to announce the first stable release of the Reef series. > >

[ceph-users] Re: Ceph Quincy and liburing.so.2 on Rocky Linux 9

Hi, In most cases the 'Alternative' distro like Alma or Rocky have outdated versions of packages, if we compared it with CentOS Stream 8 or CentOS Stream 9. For example is a golang package, on c8s is a 1.20 version on Alma still 1.19 You can try to use c8s/c9s or try to contribute to your

[ceph-users] Re: Luminous Bluestore issues and RGW Multi-site Recovery

Hi, Can you show `smartctl -a` for this device? This drives show input/output errors in dmesg when you try to run ceph-osd? k Sent from my iPhone > On 2 Aug 2023, at 21:44, Greg O'Neill wrote: > > Syslog says the drive is not in write-protect mode, however smart says life > remaining is at

[ceph-users] Re: Luminous Bluestore issues and RGW Multi-site Recovery

Hi > > On 31 Jul 2023, at 03:12, Gregory O'Neill wrote: > > /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0 > == "unexpected aio error") The Kernel was returned error actually. Try look to dmesg, may be your SATA SSD in read only mode k Sent from my iPhone

[ceph-users] Re: cephadm and kernel memory usage

Without determining what exactly process (kernel or userspace) "eat" memory, the ceph-users can't tell what exactly use memory, because don't see your display with your eyes  You should run this commands on good & bad hosts to see the real difference. This may be related to kernel version, or

[ceph-users] Re: cephadm and kernel memory usage

Good, > On 24 Jul 2023, at 20:01, Luis Domingues wrote: > > Of course: > > free -h > totalusedfree shared buff/cache available > Mem: 125Gi96Gi 9.8Gi 4.0Gi19Gi > 7.6Gi > Swap:0B 0B 0B

[ceph-users] Re: Does ceph permit the definition of new classes?

Hi, You definitely can add any other class name k Sent from my iPhone > On 24 Jul 2023, at 16:04, wodel youchi wrote: > > Can I define new device classes in ceph, I know that there are hdd, ssd and > nvme, but can I define other classes? ___

[ceph-users] Re: cephadm and kernel memory usage

Hi, Can you paste `free -h` output for this hosts? k Sent from my iPhone > On 24 Jul 2023, at 14:42, Luis Domingues wrote: > > Hi, > > So after, looking into OSDs memory usage, which seem to be fine, on a > v16.2.13 running with cephadm, on el8, it seems that the kernel is using a > lot

[ceph-users] Re: librbd hangs during large backfill

Hi, Check you libvirt limits for qemu open files/sockets. Seems, when you added new OSD's, your librbd client limit reached k Sent from my iPhone > On 18 Jul 2023, at 19:32, Wesley Dillingham wrote: > > Did your automation / process allow for stalls in between changes to allow > peering to

[ceph-users] Re: Another Pacific point release?

Hi, > On 17 Jul 2023, at 12:53, Ponnuvel Palaniyappan wrote: > > The typical EOL date (2023-06-01) has already passed for Pacific. Just > wondering if there's going to be another Pacific point release (16.2.14) in > the pipeline. Good point! At least, for possibility upgrade RBD clusters from

[ceph-users] Re: MON sync time depends on outage duration

This is a guess, the databases is like to swell. Especially the Level DB's, can grow x2 and reduce tens of percent of total size. This may be just another SST file creation, 1GB by default, Ii I remember it right Do you was looks to Grafana, about this HDD's utilization, IOPS? k Sent from my

[ceph-users] Re: MON sync time depends on outage duration

Hi, And additional to Dan suggestion, the HDD is not a good choices for RocksDB, which is most likely the reason for this thread, I think that from the 3rd time the database just goes into compaction maintenance k Sent from my iPhone > On 6 Jul 2023, at 23:48, Eugen Block wrote: > The MON

[ceph-users] Re: ceph quota qustion

Hi, This is incomplete multiparts I guess, you should remove it first. Don't know how S3 Browser works with this entities k Sent from my iPhone > On 6 Jul 2023, at 07:57, sejun21@samsung.com wrote: > > Hi, I contact you for some question about quota. > > Situation is following below. >

[ceph-users] Re: Get bucket placement target

Hi, > On 3 Jul 2023, at 12:23, mahnoosh shahidi wrote: > > So clients can not get the value which they set in the LocationConstraint > field in the create bucket request as in this doc > ? LocationConstraint in this case

[ceph-users] Re: Get bucket placement target

Hi, > On 2 Jul 2023, at 17:17, mahnoosh shahidi wrote: > > Is there any way for clients (without rgw-admin access) to get the > placement target of their S3 buckets? The "GetBucketLocation'' api returns > "default" for all placement targets and I couldn't find any other S3 api > for this

[ceph-users] Re: Applying crush rule to existing live pool

Hi, > On 27 Jun 2023, at 18:31, Rok Jaklič wrote: > > I want to place an existing pool with data to ssd-s. > > I've created crush rule: > ceph osd crush rule create-replicated replicated_ssd default host ssd > > If I apply this rule to the existing pool default.rgw.buckets.index with > 180G

[ceph-users] Re: rbd ls failed with operation not permitted

Hi, > On 7 Jun 2023, at 14:39, zyz wrote: > > When set the user's auth and then ls namespace, it is ok. > > > But when I set the user's auth with namespace, ls namespace returns with > error, but why? Because data with namespaces in "without namespace" space k

[ceph-users] Re: The pg_num from 1024 reduce to 32 spend much time, is there way to shorten the time？

Hi, > On 7 Jun 2023, at 10:02, Louis Koo wrote: > > I had set it from 0.05 to 1 with "ceph config set mon > target_max_misplaced_ratio 1.0", it's still invalid. Because is setting for a mgr, not for mon, try `ceph config set mgr target_max_misplaced_ratio 1` Cheers, k

[ceph-users] Re: Duplicate help statements in Prometheus metrics in 16.2.13

Hi Andreas, > On 5 Jun 2023, at 14:57, Andreas Haupt wrote: > > after the update to CEPH 16.2.13 the Prometheus exporter is wrongly > exporting multiple metric help & type lines for ceph_pg_objects_repaired: > > [mon1] /root #curl -sS http://localhost:9283/metrics > # HELP

[ceph-users] Re: Cluster without messenger v1, new MON still binds to port 6789

Hi, > On 1 Jun 2023, at 12:50, Robert Sander wrote: > > a cluster has ms_bind_msgr1 set to false in the config database. > > Newly created MONs still listen on port 6789 and add themselves as providing > messenger v1 into the monmap. > > How do I change that? > > Shouldn't the MONs use the

[ceph-users] Re: BlueStore fragmentation woes

Hi Igor, Should we backpot this to the p,q and reef release's? Thanks, k Sent from my iPhone > On 25 May 2023, at 23:13, Igor Fedotov wrote: > > You might be facing the issue fixed by https://github.com/ceph/ceph/pull/49885 ___ ceph-users mailing

[ceph-users] Re: cephfs max_file_size

Hi, > On 23 May 2023, at 13:27, Dietmar Rieder wrote: > > can the cephfs "max_file_size" setting be changed at any point in the > lifetime of a cephfs? > Or is it critical for existing data if it is changed after some time? Is > there anything to consider when changing, let's say, from 1TB

[ceph-users] Re: Dedicated radosgw gateways

Hi, > On 18 May 2023, at 23:04, Rok Jaklič wrote: > > I've searched for rgw_enable_lc_threads and rgw_enable_gc_threads a bit. > > but there is little information about those settings. Is there any > documentation in the wild about those settings? This is Life Cycle (see S3 lifecycle policy

[ceph-users] Re: Discussion thread for Known Pacific Performance Regressions

Hi Mark! Thank you very much for this message, acknowledging the problem publicly is the beginning of fixing it ❤️ > On 11 May 2023, at 17:38, Mark Nelson wrote: > > Hi Everyone, > > This email was originally posted to d...@ceph.io, but Marc mentioned that he > thought this would be useful

[ceph-users] Re: Dedicated radosgw gateways

Hi, > On 15 May 2023, at 14:58, Michal Strnad wrote: > > at Cephalocon 2023, it was mentioned several times that for service tasks > such as data deletion via garbage collection or data replication in S3 via > zoning, it is good to do them on dedicated radosgw gateways and not mix them >

[ceph-users] Re: CEPH Version choice

Hi, > On 15 May 2023, at 11:37, Tino Todino wrote: > > What are the main reasons for not upgrading to the latest and greatest? One of the main reasons - "just can't", because your Ceph-based products will get worse at real (not benchmark) performance, see [1] [1]

[ceph-users] Re: Ovirt integration with Ceph

Hi, Can you see logs at the vdsm.log file? What exactly happened on storage domain connection? k Sent from my iPhone > On 26 Apr 2023, at 00:37, kushagra.gu...@hsc.com wrote: > > Hi Team, > > We are trying to integrate ceph with ovirt. > We have deployed ovirt 4.4. > We want to create a

[ceph-users] Re: deploying Ceph using FQDN for MON / MDS Services

Hi, Just add POSIX domain, fstype ceph This is equivalent of mount -t Ceph on ovirt side k Sent from my iPhone > On 21 Apr 2023, at 05:24, Lokendra Rathour wrote: > > Hi Robert / Team, > Further we are now trying to integrate the ceph as storage domain in OVirt > 4.4 > > > We want to

[ceph-users] Re: pacific el7 rpms

On EL7 only Nautilus was present. Pacific was from EL8 k > On 17 Apr 2023, at 11:29, Marc wrote: > > > Is there ever going to be rpms in > > https://download.ceph.com/rpm-pacific/el7/ ___ ceph-users mailing list -- ceph-users@ceph.io To

[ceph-users] Re: deploying Ceph using FQDN for MON / MDS Services

Hi, This by the reason of DNS. Something from userland should be provide IP addresses for kernel k Sent from my iPhone > On 17 Apr 2023, at 05:56, Lokendra Rathour wrote: > > Hi Team, > The mount at the client side should be independent of Ceph, but here in > this case of DNS SRV-based

[ceph-users] Re: OSDs remain not in after update to v17

Hi, This PR for the main branch and was never backpoted to another branches, currently k Sent from my iPhone > On 15 Apr 2023, at 21:00, Alexandre Becholey wrote: > > Hi, > > Thank you for your answer, yes this seems to be exactly my issue. The pull > request related to the issue is this

[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...

Hi, Current controller mode is RAID. You can switch to HBA mode and disable cache in controller settings at the BIOS k Sent from my iPhone > On 15 Apr 2023, at 12:11, Marco Gaiarin wrote: > > Mandi! Frank Schilder > In chel di` si favelave... > >>> iops: min=2, max= 40,

[ceph-users] Re: monitoring apply_latency / commit_latency ?

 Hi, > On 2 Apr 2023, at 23:14, Matthias Ferdinand wrote: > > I understand that grafana graphs are generated from prometheus metrics. > I just wanted to know which OSD daemon-perf values feed these prometheus > metrics (or if they are generated in some other way). Yep, this perf metrics is

[ceph-users] Re: monitoring apply_latency / commit_latency ?

Hi, > On 25 Mar 2023, at 23:15, Matthias Ferdinand wrote: > > from "ceph daemon osd.X perf dump"? No, from ceph-mgr prometheus exporter You can enable it via `ceph mgr module enable prometheus` > Please bear with me :-) I just try to get some rough understanding what > the numbers to be

[ceph-users] Re: monitoring apply_latency / commit_latency ?

Hi Matthias, Prometheus exporter already have all this metrics, you can setup Grafana panels as you want Also, the apply latency in a metric for a pre-bluestore, i.e. filestore For Bluestore apply latency is the same as commit latency, you can check this via `ceph osd perf` command k > On

[ceph-users] Re: avg apply latency went up after update from octopus to pacific

Hi, Maybe [1] ? [1] https://tracker.ceph.com/issues/58530 k > On 22 Mar 2023, at 16:20, Boris Behrens wrote: > > Are there any other ides? > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to

[ceph-users] Re: s3 compatible interface

Hi, You can try [1] geesefs project, the presentation for this code is here [2] [1] https://github.com/yandex-cloud/geesefs [2] https://yourcmc-ru.translate.goog/geesefs-2022/highload.html?_x_tr_sl=ru&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp k > On 28 Feb 2023, at 22:31, Marc wrote: > >

[ceph-users] Re: ceph noout vs ceph norebalance, which is better for minor maintenance

> On 17 Feb 2023, at 23:20, Anthony D'Atri wrote: > > > >> * if rebalance will starts due EDAC or SFP degradation, is faster to fix the >> issue via DC engineers and put node back to work > > A judicious mon_osd_down_out_subtree_limit setting can also do this by not > rebalancing when an

[ceph-users] Re: ceph noout vs ceph norebalance, which is better for minor maintenance

> On 17 Feb 2023, at 23:20, Anthony D'Atri wrote: > > > >> * if rebalance will starts due EDAC or SFP degradation, is faster to fix the >> issue via DC engineers and put node back to work > > A judicious mon_osd_down_out_subtree_limit setting can also do this by not > rebalancing when an

[ceph-users] Re: ceph noout vs ceph norebalance, which is better for minor maintenance

Hi Will, All our clusters with noout flag by default, since cluster birth. The reasons: * if rebalance will starts due EDAC or SFP degradation, is faster to fix the issue via DC engineers and put node back to work * noout prevents unwanted OSD's fills and the run out of space => outage of

[ceph-users] Re: iDRAC 9 version 6.10 shows 0% for write endurance on non-dell drives, work around?

 Hi, You can use smartctl_exporter [1] for all your media, not only the SSD k [1] https://github.com/prometheus-community/smartctl_exporter Sent from my iPhone > On 14 Feb 2023, at 23:01, Drew Weaver wrote: > Hello, > > After upgrading a lot of iDRAC9 modules to version 6.10 in servers

[ceph-users] Re: Ceph Quincy On Rocky 8.x - Upgrade To Rocky 9.1

___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Quincy On Rocky 8.x - Upgrade To Rocky 9.1

Hi, Seems packages el9_quincy are available [1] You can try k [1] https://download.ceph.com/rpm-quincy/el9/x86_64/ > On 10 Feb 2023, at 13:23, duluxoz wrote: > > Sorry if this was mentioned previously (I obviously missed it if it was) but > can we upgrade a Ceph Quincy Host/Cluster from

[ceph-users] Re: January Ceph Science Virtual User Group

Hi Mark, Why you need to upgrade every year? Yes, scalable - you can add new racks without new version or new distro - that how big clusters are live k Sent from my iPhone > On 2 Feb 2023, at 19:09, Marc wrote: > >  >> >> >>

[ceph-users] Re: cephadm upgrade 16.2.10 to 16.2.11: osds crash and get stuck restarting

Hi Zakhar, > On 26 Jan 2023, at 08:33, Zakhar Kirpichenko wrote: > > Jan 25 23:07:53 ceph01 bash[2553123]: >

[ceph-users] Re: Status of Quincy 17.2.5 ?

May be Mike can organize this release flow...  CC'ed Mike Perez, I think team need some manager observability (a little) k > On 25 Jan 2023, at 16:26, Christian Rohmann > wrote: > > Hey everyone, > > > On 20/10/2022 10:12, Christian Rohmann wrote: >> 1) May I bring up again my remarks

[ceph-users] Re: ceph cluster iops low

Hi, You SSD is a "desktop" SSD, not a "enterprise" SSD, see [1] This mostly was't suitable for Ceph [1] https://yourcmc.ru/wiki/Ceph_performance#CAPACITORS.21 k > On 25 Jan 2023, at 05:35, peter...@raksmart.com wrote: > > Hi Mark, > Thanks for your response, it is help! > Our Ceph cluster

[ceph-users] Re: Mount ceph using FQDN

Hi, Do you think kernel should care about DNS resolution? k > On 24 Jan 2023, at 19:07, kushagra.gu...@hsc.com wrote: > > Hi team, > > We have a ceph cluster with 3 storage nodes: > 1. storagenode1 - abcd:abcd:abcd::21 > 2. storagenode2 - abcd:abcd:abcd::22 > 3. storagenode3 -

[ceph-users] Re: Current min_alloc_size of OSD?

Hi, > On 12 Jan 2023, at 04:35, Robert Sander wrote: > > How can I get the current min_allloc_size of OSDs that were created with > older Ceph versions? Is there a command that shows this info from the on disk > format of a bluestore OSD? You can see this via kvstore-tool:

[ceph-users] Re: [ERR] OSD_SCRUB_ERRORS: 2 scrub errors

Hi, > On 10 Jan 2023, at 07:10, David Orman wrote: > > We ship all of this to our centralized monitoring system (and a lot more) and > have dashboards/proactive monitoring/alerting with 100PiB+ of Ceph. If you're > running Ceph in production, I believe host-level monitoring is critical, >

[ceph-users] Re: max pool size (amount of data/number of OSDs)

Hi Chris, The actually limits are not software. Usually Ceph teams on Cloud Providers or Universities running out at physical resources at first: racks, racks power or network (ports, EOL switches that can't be upgraded) or hardware lifetime (There is no point in buying old hardware, and the

[ceph-users] Re: radosgw not working after upgrade to Quincy

Hi, Just try to read your logs: > 2022-12-29T02:07:38.953+ 7f5df868ccc0 0 WARNING: skipping unknown > framework: civetweb You try to use the `civetweb`, it was absent in quincy release. You need to update your configs and use `beast` instead k > On 29 Dec 2022, at 09:20, Andrei

[ceph-users] Re: Does Replica Count Affect Tell Bench Result or Not?

Hi, The cache was gone, optimize is proceed. This is not enterprise device, you should never use it with Ceph  k Sent from my iPhone > On 27 Dec 2022, at 16:41, hosseinz8...@yahoo.com wrote: > >  Thanks AnthonyI have a cluster with QLC SSD disks (Samsung QVO 860). The > cluster works for

[ceph-users] Re: Empty /var/lib/ceph/osd/ceph-$osd after reboot

Hi, ceph-volume lvm activate --all k > On 21 Dec 2022, at 13:53, Isaiah Tang Yue Shun wrote: > > From what I understand, after creating an OSD using "ceph-volume lvm > create", we will do a "ceph-volume lvm activate" so that the systemd is > created. > > However, I found that after

[ceph-users] Re: Quincy 22.04/Jammy packages

Thank you Ilya! > On 21 Oct 2022, at 21:02, Ilya Dryomov wrote: > > On Fri, Oct 21, 2022 at 12:48 PM Konstantin Shalygin wrote: >> >> CC'ed David > > Hi Konstantin, > > David has decided to pursue something else and is no longer working on > Ceph [1].

[ceph-users] Re: Quincy 22.04/Jammy packages

CC'ed David Maybe Ilya can tag someone from DevOps additionally Thanks, k > On 20 Oct 2022, at 20:07, Goutham Pacha Ravi wrote: > > +1 > The OpenStack community is interested in this as well. We're trying to move > all our ubuntu testing to Ubuntu Jammy/22.04 [1]; and we consume packages >

[ceph-users] Re: How to remove remaining bucket index shard objects

This strange stats, at least one object should be exists for this OMAP's. Try to deep-scrub this PG, try to list objects in this PG `rados ls --pgid 6.2` k Sent from my iPhone > On 18 Oct 2022, at 03:39, Yuji Ito wrote: > > Thank you for your reply. > >> the object need only for OMAP

  1   2   3   4   >