[ceph-users] Re: Multisite: metadata behind on shards

2024-05-13 Thread Christian Rohmann
versions. But there have been lots of fixes in this area ... e.g. https://tracker.ceph.com/issues/39657 Is upgrading Ceph to a more recent version an option for you? Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

2024-04-19 Thread Christian Rohmann
so it's sure to be picked up. Thanks a bunch. If you miss the train, you miss the train - fair enough. Nice to know there is another one going soon and that bug is going to be on it ! Regards Christian ___ ceph-users mailing list -- ceph-users

[ceph-users] Re: reef 18.2.3 QE validation status

2024-04-18 Thread Christian Rohmann
in one my clusters. Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: rgw s3 bucket policies limitations (on users)

2024-04-03 Thread Christian Rohmann
. I would love for RGW to support more detailed bucket policies, especially with external / Keystone authentication. Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Hanging request in S3

2024-03-12 Thread Christian Kugler
Hi Casey, Interesting. Especially since the request it hangs on is a GET request. I set the option and restarted the RGW I test with. The POSTs for deleting take a while but there are not longer blocking GET or POST requests. Thank you! Best, Christian PS: Sorry for pressing the wrong reply

[ceph-users] Re: Journal size recommendations

2024-03-08 Thread Christian Rohmann
"This section applies only to the older Filestore OSD back end. Since Luminous BlueStore has been default and preferred." It's totally obsolete with bluestore. Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To u

[ceph-users] Re: rgw dynamic bucket sharding will hang io

2024-03-08 Thread Christian Rohmann
On 08.03.24 14:25, Christian Rohmann wrote: What do you mean by blocking IO? No bucket actions (read / write) or high IO utilization? According to https://docs.ceph.com/en/latest/radosgw/dynamicresharding/ "Writes to the target bucket are blocked (but reads are not) briefly during resha

[ceph-users] Re: rgw dynamic bucket sharding will hang io

2024-03-08 Thread Christian Rohmann
you mean by blocking IO? No bucket actions (read / write) or high IO utilization? Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Hanging request in S3

2024-03-06 Thread Christian Kugler
situation or at least where to or what to look for? Best, Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: debian-reef_OLD?

2024-03-05 Thread Christian Rohmann
On 04.03.24 22:24, Daniel Brown wrote: debian-reef/ Now appears to be: debian-reef_OLD/ Could this have been  some sort of "release script" just messing up the renaming / symlinking to the most recent stable? Regards Christian ___

[ceph-users] Re: ceph-crash NOT reporting crashes due to wrong permissions on /var/lib/ceph/crash/posted (Debian / Ubuntu packages)

2024-02-29 Thread Christian Rohmann
On 23.02.24 16:18, Christian Rohmann wrote: I just noticed issues with ceph-crash using the Debian /Ubuntu packages (package: ceph-base): While the /var/lib/ceph/crash/posted folder is created by the package install, it's not properly chowned to ceph:ceph by the postinst script

[ceph-users] ceph-crash NOT reporting crashes due to wrong permissions on /var/lib/ceph/crash/posted (Debian / Ubuntu packages)

2024-02-23 Thread Christian Rohmann
. Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Throughput metrics missing iwhen updating Ceph Quincy to Reef

2024-02-05 Thread Christian Rohmann
On 01.02.24 10:10, Christian Rohmann wrote: [...] I am wondering if ceph-exporter ([2] is also built and packaged via the ceph packages [3] for installations that use them? [2] https://github.com/ceph/ceph/tree/main/src/exporter [3] https://docs.ceph.com/en/latest/install/get-packages/ I

[ceph-users] Re: how can install latest dev release?

2024-02-01 Thread Christian Rohmann
atest" documentation is at https://docs.ceph.com/en/latest/install/get-packages/#ceph-development-packages. But it seems nothing has changed. There are dev packages available at the URLs mentioned there. Regards Christian ___ ceph-users mailing li

[ceph-users] Re: Throughput metrics missing iwhen updating Ceph Quincy to Reef

2024-02-01 Thread Christian Rohmann
wondering if ceph-exporter ([2] is also built and packaged via the ceph packages [3] for installations that use them? Regards Christian [1] https://docs.ceph.com/en/latest/mgr/prometheus/#ceph-daemon-performance-counters-metrics [2] https://github.com/ceph/ceph/tree/main/src/exporter [3

[ceph-users] Re: how can install latest dev release?

2024-01-31 Thread Christian Rohmann
containers being built somewhere to use with cephadm. Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 3 DC with 4+5 EC not quite working

2024-01-14 Thread Christian Wuerdig
I could be wrong however as far as I can see you have 9 chunks which requires 9 failure domains. Your failure domain is set to datacenter which you only have 3 of. So that won't work. You need to set your failure domain to host and then create a crush rule to choose a DC and choose 3 hosts within

[ceph-users] Re: RGW rate-limiting or anti-hammering for (external) auth requests // Anti-DoS measures

2024-01-12 Thread Christian Rohmann
(Keystone in my case) at full rate. Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW rate-limiting or anti-hammering for (external) auth requests // Anti-DoS measures

2024-01-09 Thread Christian Rohmann
Happy New Year Ceph-Users! With the holidays and people likely being away, I take the liberty to bluntly BUMP this question about protecting RGW from DoS below: On 22.12.23 10:24, Christian Rohmann wrote: Hey Ceph-Users, RGW does have options [1] to rate limit ops or bandwidth per bucket

[ceph-users] Re: cephadm - podman vs docker

2023-12-31 Thread Christian Wuerdig
General complaint about docker is usually that it by default stops all running containers when the docker daemon gets shutdown. There is the "live-restore" option (which has been around for a while) but that's turned off by default (and requires a daemon restart to enable). It only supports patch

[ceph-users] RGW rate-limiting or anti-hammering for (external) auth requests // Anti-DoS measures

2023-12-22 Thread Christian Rohmann
lace? * Does it make sense to extend RGWs capabilities to deal with those cases itself? ** adding negative caching ** rate limits on concurrent external authentication requests (or is there a pool of connections for those requests?) Regards Christian [1] https://docs.ceph.com/en/latest

[ceph-users] Re: EC Profiles & DR

2023-12-05 Thread Christian Wuerdig
You can structure your crush map so that you get multiple EC chunks per host in a way that you can still survive a host outage outage even though you have fewer hosts than k+1 For example if you run an EC=4+2 profile on 3 hosts you can structure your crushmap so that you have 2 chunks per host.

[ceph-users] Automatic triggering of the Ubuntu SRU process, e.g. for the recent 17.2.7 Quincy point release?

2023-11-12 Thread Christian Rohmann
Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Packages for 17.2.7 released without release notes / announcement (Re: Re: Status of Quincy 17.2.5 ?)

2023-10-30 Thread Christian Rohmann
Sorry to dig up this old thread ... On 25.01.23 10:26, Christian Rohmann wrote: On 20/10/2022 10:12, Christian Rohmann wrote: 1) May I bring up again my remarks about the timing: On 19/10/2022 11:46, Christian Rohmann wrote: I believe the upload of a new release to the repo prior

[ceph-users] Re: Hardware recommendations for a Ceph cluster

2023-10-10 Thread Christian Wuerdig
On Mon, 9 Oct 2023 at 14:24, Anthony D'Atri wrote: > > > > AFAIK the standing recommendation for all flash setups is to prefer fewer > > but faster cores > > Hrm, I think this might depend on what you’re solving for. This is the > conventional wisdom for MDS for sure. My sense is that OSDs can

[ceph-users] Re: Hardware recommendations for a Ceph cluster

2023-10-08 Thread Christian Wuerdig
AFAIK the standing recommendation for all flash setups is to prefer fewer but faster cores, so something like a 75F3 might be yielding better latency. Plus you probably want to experiment with partitioning the NVMEs and running multiple OSDs per drive - either 2 or 4. On Sat, 7 Oct 2023 at 08:23,

[ceph-users] CVE-2023-43040 - Improperly verified POST keys in Ceph RGW?

2023-09-27 Thread Christian Rohmann
) about this, as I could not find one yet. It seems a weird way of disclosing such a thing and am wondering if anybody knew any more about this? Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email

[ceph-users] What is causing *.rgw.log pool to fill up / not be expired (Re: RGW multisite logs (data, md, bilog) not being trimmed automatically?)

2023-09-14 Thread Christian Rohmann
I am unfortunately still observing this issue of the RADOS pool "*.rgw.log" filling up with more and more objects: On 26.06.23 18:18, Christian Rohmann wrote: On the primary cluster I am observing an ever growing (objects and bytes) "sitea.rgw.log" pool, not so on the r

[ceph-users] Re: Contionuous spurious repairs without cause?

2023-09-06 Thread Christian Theune
Hi, interesting, that’s something we can definitely try! Thanks! Christian > On 5. Sep 2023, at 16:37, Manuel Lausch wrote: > > Hi, > > in older versions of ceph with the auto-repair feature the PG state of > scrubbing PGs had always the repair state as well. > With la

[ceph-users] Re: Contionuous spurious repairs without cause?

2023-09-06 Thread Christian Theune
all daemons to the same minor version those > errors were gone. > > Regards, > Eugen > > Zitat von Christian Theune : > >> Hi, >> >> this is a bit older cluster (Nautilus, bluestore only). >> >> We’ve noticed that the cluster is almost continuous

[ceph-users] Contionuous spurious repairs without cause?

2023-09-05 Thread Christian Theune
any relevant issue either. Any ideas? Liebe Grüße, Christian Theune -- Christian Theune · c...@flyingcircus.io · +49 345 219401 0 Flying Circus Internet Operations GmbH · https://flyingcircus.io Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland HR Stendal HRB 21169 · Geschäftsführer

[ceph-users] Re: Can ceph-volume manage the LVs optionally used for DB / WAL at all?

2023-08-26 Thread Christian Rohmann
his is what I am currently doing (lvcreate + ceph-volume lvm create). My question therefore is, if ceph-volume (!) could somehow create this LV for the DB automagically if I'd just give it a device (or existing VG)? Thank you very much for your patience in clarifying and responding to my

[ceph-users] Re: Can ceph-volume manage the LVs optionally used for DB / WAL at all?

2023-08-25 Thread Christian Rohmann
with DB or WAL on fast storage. Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] When to use the auth profiles simple-rados-client and profile simple-rados-client-with-blocklist?

2023-08-22 Thread Christian Rohmann
name like the rbd and the corresponding "rbd-read-only" profile? Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Can ceph-volume manage the LVs optionally used for DB / WAL at all?

2023-08-11 Thread Christian Rohmann
a few LVs is hard... it's just that ceph volume does apply some structure to the naming of LVM VGs and LVs on the OSD device and also adds metadata. That would then be up to the user, right? Regards Christian ___ ceph-users mailing list -- ceph-use

[ceph-users] Re: ceph-volume lvm new-db fails

2023-08-11 Thread Christian Rohmann
On 10/08/2023 13:30, Christian Rohmann wrote: It's already fixed master, but the backports are all still pending ... There are PRs for the backports now: * https://tracker.ceph.com/issues/62060 * https://tracker.ceph.com/issues/62061 * https://tracker.ceph.com/issues/62062 Regards

[ceph-users] Re: ceph-volume lvm new-db fails

2023-08-10 Thread Christian Rohmann
: https://tracker.ceph.com/issues/55260 It's already fixed master, but the backports are all still pending ... Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Not all Bucket Shards being used

2023-08-02 Thread Christian Kugler
> Thank you for the information, Christian. When you reshard the bucket id is > updated (with most recent versions of ceph, a generation number is > incremented). The first bucket id matches the bucket marker, but after the > first reshard they diverge. This makes a lot of sense

[ceph-users] Re: Not all Bucket Shards being used

2023-07-25 Thread Christian Kugler
_info": "false" } } > 4. After you resharded previously, did you get command-line output along the > lines of: > 2023-07-24T13:33:50.867-0400 7f10359f2a80 1 execute INFO: reshard of bucket > “" completed successfully I think so, at least for the second reshard. But I wouldn't bet my life on it. I fear I might have missed an error on the first one since I have done a radosgw-admin bucket reshard so often and never seen it fail. Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Not all Bucket Shards being used

2023-07-18 Thread Christian Kugler
something like 97. Or I could directly "downshard" to 97. Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone? Best, Christian Kugler ___

[ceph-users] Re: Adding datacenter level to CRUSH tree causes rebalancing

2023-07-16 Thread Christian Wuerdig
Based on my understanding of CRUSH it basically works down the hierarchy and then randomly (but deterministically for a given CRUSH map) picks buckets (based on the specific selection rule) on that level for the object and then it does this recursively until it ends up at the leaf nodes. Given

[ceph-users] Re: RGW accessing real source IP address of a client (e.g. in S3 bucket policies)

2023-07-06 Thread Christian Rohmann
. In reality it was simply the private, RFC1918, IP of the test machine that came in as source. Sorry for the noise and thanks for your help. Christian P.S. With IPv6, this would not have happened. ___ ceph-users mailing list -- ceph-users@ceph.io

[ceph-users] Re: RGW multisite logs (data, md, bilog) not being trimmed automatically?

2023-06-29 Thread Christian Rohmann
is that required and why seems to be no periodic trimming happening? Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Bluestore compression - Which algo to choose? Zstd really still that bad?

2023-06-27 Thread Christian Rohmann
looking for data other might have collected on their similar use-cases. Also I am still wondering if there really is nobody that worked/played more with zstd since that has become so popular in recent months... Regards Christian ___ ceph-users

[ceph-users] RGW multisite logs (data, md, bilog) not being trimmed automatically?

2023-06-26 Thread Christian Rohmann
g of the log trimming activity that I should expect? Or that might indicate why trimming does not happen? Regards Christian [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/WZCFOAMLWV3XCGJ3TVLHGMJFVYNZNKLD/ ___ ceph-users mail

[ceph-users] Re: Radogw ignoring HTTP_X_FORWARDED_FOR header

2023-06-26 Thread Christian Rohmann
ot;bytes_sent":0,"bytes_received":64413,"object_size":64413,"total_time":155,"user_agent":"aws-sdk-go/1.27.0 (go1.16.15; linux; amd64) S3Manager","referrer":"","trans_id":"REDACTED","authentication_typ

[ceph-users] Bluestore compression - Which algo to choose? Zstd really still that bad?

2023-06-26 Thread Christian Rohmann
e decision on the compression algo? Regards Christian [1] https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#confval-bluestore_compression_algorithm [2] https://github.com/ceph/ceph/pull/33790 [3] https://github.com/facebook/zstd/

[ceph-users] ceph quincy repo update to debian bookworm...?

2023-06-22 Thread Christian Peters
://download.coeh.com/debian-quincy/ bullseye main to deb https://download.coeh.com/debian-quincy/ boowkworm main in the near future!? Regards, Christian OpenPGP_0xC20C05037880471C.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature

[ceph-users] Re: RGW: Migrating a long-lived cluster to multi-site, fixing an EC pool mistake

2023-06-21 Thread Christian Theune
zonegroups referring to the same pools and this should only run through proper abstractions … o_O Cheers, Christian > On 14. Jun 2023, at 17:42, Christian Theune wrote: > > Hi, > > further note to self and for posterity … ;) > > This turned out to be a no-go as well, becau

[ceph-users] Re: RGW: Migrating a long-lived cluster to multi-site, fixing an EC pool mistake

2023-06-16 Thread Christian Theune
id i get something wrong? > > > > > Kind regards, > Nino > > > On Wed, Jun 14, 2023 at 5:44 PM Christian Theune wrote: > Hi, > > further note to self and for posterity … ;) > > This turned out to be a no-go as well, because you can’t silently switch th

[ceph-users] Re: RGW accessing real source IP address of a client (e.g. in S3 bucket policies)

2023-06-16 Thread Christian Rohmann
, not the public IP of the client. So the actual remote address is NOT used in my case. Did I miss any config setting anywhere? Regards and thanks for your help Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] RGW accessing real source IP address of a client (e.g. in S3 bucket policies)

2023-06-15 Thread Christian Rohmann
ately seems not even supposed by the BEAST library which RGW uses.     I opened feature requests ...      ** https://tracker.ceph.com/issues/59422      ** https://github.com/chriskohlhoff/asio/issues/1091      ** https://github.com/boostorg/beast/issues/2484    but there is no outcome yet. Rega

[ceph-users] Re: RGW: Migrating a long-lived cluster to multi-site, fixing an EC pool mistake

2023-06-14 Thread Christian Theune
a few very large buckets (200T+) that will take a while to copy. We can pre-sync them of course, so the downtime will only be during the second copy. Christian > On 13. Jun 2023, at 14:52, Christian Theune wrote: > > Following up to myself and for posterity: > > I’m going to

[ceph-users] Re: RGW: Migrating a long-lived cluster to multi-site, fixing an EC pool mistake

2023-06-13 Thread Christian Theune
is still 2.4 hours … Cheers, Christian > On 9. Jun 2023, at 11:16, Christian Theune wrote: > > Hi, > > we are running a cluster that has been alive for a long time and we tread > carefully regarding updates. We are still a bit lagging and our cluster (that > started around

[ceph-users] RGW: Migrating a long-lived cluster to multi-site, fixing an EC pool mistake

2023-06-09 Thread Christian Theune
and I guess that would be a good comparison for what timing to expect when running an update on the metadata. I’ll also be in touch with colleagues from Heinlein and 42on but I’m open to other suggestions. Hugs, Christian [1] We currently have 215TiB data in 230M objects. Using the “official

[ceph-users] Re: Encryption per user Howto

2023-05-22 Thread Christian Wuerdig
Hm, this thread is confusing in the context of S3 client-side encryption means - the user is responsible to encrypt the data with their own keys before submitting it. As far as I'm aware, client-side encryption doesn't require any specific server support - it's a function of the client SDK used

[ceph-users] Re: pg_autoscaler using uncompressed bytes as pool current total_bytes triggering false POOL_TARGET_SIZE_BYTES_OVERCOMMITTED warnings?

2023-04-21 Thread Christian Rohmann
enlighten me. Thank you and with kind regards Christian On 02/02/2022 20:10, Christian Rohmann wrote: Hey ceph-users, I am debugging a mgr pg_autoscaler WARN which states a target_size_bytes on a pool would overcommit the available storage. There is only one pool with value

[ceph-users] Re: Eccessive occupation of small OSDs

2023-04-02 Thread Christian Wuerdig
With failure domain host your max usable cluster capacity is essentially constrained by the total capacity of the smallest host which is 8TB if I read the output correctly. You need to balance your hosts better by swapping drives. On Fri, 31 Mar 2023 at 03:34, Nicola Mori wrote: > Dear Ceph

[ceph-users] External Auth (AssumeRoleWithWebIdentity) , STS by default, generic policies and isolation by ownership

2023-03-15 Thread Christian Rohmann
create their own roles and policies to use them by default? All the examples talk about the requirement for admin caps and individual setting of '--caps="user-policy=*'. If there was a default role + policy (question #1) that could be applied to externally authenticated users, I'd like for

[ceph-users] Re: Trying to throttle global backfill

2023-03-09 Thread Rice, Christian
ative of the community response. I learned a lot in the process, had an outage-inducing scenario rectified very quickly, and got back to work. Thanks so much! Happy to answer any followup questions and return the favor when I can. From: Rice, Christian Date: Wednesday, March 8, 2023 at 3:57 PM To:

[ceph-users] Trying to throttle global backfill

2023-03-08 Thread Rice, Christian
I have a large number of misplaced objects, and I have all osd settings to “1” already: sudo ceph tell osd.\* injectargs '--osd_max_backfills=1 --osd_recovery_max_active=1 --osd_recovery_op_priority=1' How can I slow it down even more? The cluster is too large, it’s impacting other network

[ceph-users] Re: [EXTERNAL] Re: Renaming a ceph node

2023-02-15 Thread Rice, Christian
ing it with the new name. > You only must keep the ID from the node in the crushmap! > > Regards > Manuel > > > On Mon, 13 Feb 2023 22:22:35 + > "Rice, Christian" wrote: > >> Can anyone please point me at a doc that explains the most >> efficie

[ceph-users] Renaming a ceph node

2023-02-13 Thread Rice, Christian
Can anyone please point me at a doc that explains the most efficient procedure to rename a ceph node WITHOUT causing a massive misplaced objects churn? When my node came up with a new name, it properly joined the cluster and owned the OSDs, but the original node with no devices remained. I

[ceph-users] Re: Status of Quincy 17.2.5 ?

2023-01-25 Thread Christian Rohmann
Hey everyone, On 20/10/2022 10:12, Christian Rohmann wrote: 1) May I bring up again my remarks about the timing: On 19/10/2022 11:46, Christian Rohmann wrote: I believe the upload of a new release to the repo prior to the announcement happens quite regularly - it might just be due

[ceph-users] Re: OSD slow ops warning not clearing after OSD down

2023-01-16 Thread Christian Rohmann
and then a total failure of an OSD ? Would be nice to fix this though to not "block" the warning status with something that's not actually a warning. Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an ema

[ceph-users] Re: 16.2.11 branch

2022-12-15 Thread Christian Rohmann
On 15/12/2022 10:31, Christian Rohmann wrote: May I kindly ask for an update on how things are progressing? Mostly I am interested on the (persisting) implications for testing new point releases (e.g. 16.2.11) with more and more bugfixes in them. I guess I just have not looked on the right

[ceph-users] Re: 16.2.11 branch

2022-12-15 Thread Christian Rohmann
! Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] RGW Forcing buckets to be encrypted (SSE-S3) by default (via a global bucket encryption policy)?

2022-11-23 Thread Christian Rohmann
creators to apply such a policy themselves, but to apply this as a global default in RGW, forcing all buckets to have SSE enabled - transparently. If there is no way to achieve this just yet, what are your thoughts about adding such an option to RGW? Regards

[ceph-users] Re: Cloud sync to minio fails after creating the bucket

2022-11-21 Thread Christian Rohmann
I'm facing -.- But there is a fix commited, pending backports to Quincy / Pacific: https://tracker.ceph.com/issues/57306 Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cloud sync to minio fails after creating the bucket

2022-11-21 Thread Christian Rohmann
807) about Cloud Sync being broken since Pacific? Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW replication and multiple endpoints

2022-11-14 Thread Christian Rohmann
ct RGW in both zones. Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Upgrade/migrate host operating system for ceph nodes (CentOS/Rocky)

2022-11-03 Thread Prof. Dr. Christian Dietrich
Hi all, we're running a ceph cluster with v15.2.17 and cephadm on various CentOS hosts. Since CentOS 8.x is EOL, we'd like to upgrade/migrate/reinstall the OS, possibly migrating to Rocky or CentOS stream: host | CentOS | Podman -|--|--- osd* | 7.9.2009 | 1.6.4 x5 osd*

[ceph-users] Re: 16.2.11 branch

2022-10-28 Thread Christian Rohmann
which we are waiting for. TBH I was about to ask if it would not be sensible to do an intermediate release and not let it grow bigger and bigger (with even more changes / fixes)  going out at once. Regards Christian ___ ceph-users mailing list

[ceph-users] Re: Using multiple SSDs as DB

2022-10-25 Thread Christian
have) resulted in extra 8 Ceph OSDs with no db device. Best, Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Using multiple SSDs as DB

2022-10-21 Thread Christian
it sounds like it would limit the amount of SSDs used for DB devices. How can I use all of the SSDs‘ capacity? Best, Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Status of Quincy 17.2.5 ?

2022-10-20 Thread Christian Rohmann
this week. Thanks for the info. 1) May I bring up again my remarks about the timing: On 19/10/2022 11:46, Christian Rohmann wrote: I believe the upload of a new release to the repo prior to the announcement happens quite regularly - it might just be due to the technical process of releasing. But I

[ceph-users] Mirror de.ceph.com broken?

2022-10-20 Thread Christian Rohmann
debian-17.2.4/ return 404. Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Status of Quincy 17.2.5 ?

2022-10-19 Thread Christian Rohmann
date notes. Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: rgw multisite octopus - bucket can not be resharded after cancelling prior reshard process

2022-10-13 Thread Christian Rohmann
://tracker.ceph.com/projects/rgw/issues?query_id=247 But you are not syncing the data in your deployment? Maybe that's a different case then? Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le

[ceph-users] RGW multisite Cloud Sync module with support for client side encryption?

2022-09-12 Thread Christian Rohmann
es require users to actively make use of SSE-S3, right? Thanks again with kind regards, Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Suggestion to build ceph storage

2022-06-19 Thread Christian Wuerdig
On Sun, 19 Jun 2022 at 02:29, Satish Patel wrote: > Greeting folks, > > We are planning to build Ceph storage for mostly cephFS for HPC workload > and in future we are planning to expand to S3 style but that is yet to be > decided. Because we need mass storage, we bought the following HW. > > 15

[ceph-users] Re: [EXTERNAL] Laggy OSDs

2022-03-30 Thread Rice, Christian
we had issues with slow ops on ssd AND nvme; mostly fixed by raising aio-max-nr from 64K to 1M, eg "fs.aio-max-nr=1048576" if I remember correctly. On 3/29/22, 2:13 PM, "Alex Closs" wrote: Hey folks, We have a 16.2.7 cephadm cluster that's had slow ops and several (constantly

[ceph-users] Re: octopus (15.2.16) OSDs crash or don't answer heathbeats (and get marked as down)

2022-03-23 Thread Christian Wuerdig
I would not host multiple OSD on a spinning drive (unless it's one of those Seagate MACH.2 drives that have two independent heads) - head seek time will most likely kill performance. The main reason to host multiple OSD on a single SSD or NVME is typically to make use of the large IOPS capacity

[ceph-users] Re: How to clear "Too many repaired reads on 1 OSDs" on pacific

2022-03-01 Thread Christian Rohmann
On 28/02/2022 20:54, Sascha Vogt wrote: Is there a way to clear the error counter on pacific? If so, how? No, no anymore. See https://tracker.ceph.com/issues/54182 Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io

[ceph-users] Re: Random scrub errors (omap_digest_mismatch) on pgs of RADOSGW metadata pools (bug 53663)

2022-02-10 Thread Christian Rohmann
lock /semaphore or something along this line, this certainly is affected by the latency on the underlying storage. Could you maybe trigger manual a deep-scrub on all your OSDs, just to see if that does anything? Thanks again for keeping in touch! Regards Chri

[ceph-users] Re: Random scrub errors (omap_digest_mismatch) on pgs of RADOSGW metadata pools (bug 53663)

2022-02-08 Thread Christian Rohmann
are the worse of bugs and adding some unpredictability to their occurrence we likely need more evidence to have a chance to narrow this down. And since you seem to observe something similar,  could you gather and post debug info about them to the ticket as well maybe? Regards Christian

[ceph-users] Re: Random scrub errors (omap_digest_mismatch) on pgs of RADOSGW metadata pools (bug 53663)

2022-02-07 Thread Christian Rohmann
;omap_digest_mismatch","client.4349063.0:10289913"   ".dir.9cba42a3-dd1c-46d4-bdd2-ef634d12c0a5.56337947.1562","omap_digest_mismatch","client.4364800.0:10934595"   ".dir.06f9b7c7-6326-4a41-9115-d4d092cf74ce.1163207.114.9","omap_digest_mismatch&

[ceph-users] pg_autoscaler using uncompressed bytes as pool current total_bytes triggering false POOL_TARGET_SIZE_BYTES_OVERCOMMITTED warnings?

2022-02-02 Thread Christian Rohmann
rely must be a bug then, as those bytes are not really "actual_raw_used". I was about to raise a bug, but I wanted to ask here on the ML first if I misunderstood the mechanisms at play here. Thanks and with kind regards, Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Random scrub errors (omap_digest_mismatch) on pgs of RADOSGW metadata pools (bug 53663)

2021-12-21 Thread Christian Rohmann
occur on the secondary side only Regarding your scrub errors. You do have those still coming up at random? Could you check with "list-inconsistent-obj" if yours are within the OMAP data and in the metadata pools only? Regards Christian ___ ce

[ceph-users] Re: Random scrub errors (omap_digest_mismatch) on pgs of RADOSGW metadata pools (bug 53663)

2021-12-21 Thread Christian Rohmann
arge OMAP structures with lots of movement. And the issues only are with the metadata pools. Regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Random scrub errors (omap_digest_mismatch) on pgs of RADOSGW metadata pools (bug 53663)

2021-12-20 Thread Christian Rohmann
the next inconsistency occurs? Could there be any misconfiguration causing this? Thanks and with kind regards Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: inconsistent pg after upgrade nautilus to octopus

2021-12-19 Thread Christian Rohmann
repairing the other, adding to the theory of something really odd going on. Did you upgrade to Octopus in the end then? Any more issues with such inconsistencies on your side Tomasz? Regards Christian On 20/10/2021 10:33, Tomasz Płaza wrote: As the upgrade process states, rgw are the last one

[ceph-users] Re: [EXTERNAL] Re: Why you might want packages not containers for Ceph deployments

2021-11-18 Thread Christian Wuerdig
I think Marc uses containers - but they've chosen Apache Mesos as orchestrator and ceph-adm doesn't work with that. Currently essentially two ceph container orchestrators exist - rook which is a ceph orch or kubernetes and ceph-adm which is an orchestrator expecting docker or podman Admittedly I

[ceph-users] Re: Question if WAL/block.db partition will benefit us

2021-11-08 Thread Christian Wuerdig
In addition to what the others said - generally there is little point in splitting block and wal partitions - just stick to one for both. What model are you SSDs and how well do they handle small direct writes? Because that's what you'll be getting on them and the wrong type of SSD can make things

[ceph-users] Re: [Ceph] Recovery is very Slow

2021-10-28 Thread Christian Wuerdig
wrote: >> >> Den tors 28 okt. 2021 kl 10:18 skrev Lokendra Rathour >> : >> > >> > Hi Christian, >> > Thanks for the update. >> > I have 5 SSD on each node i.e. a total of 15 SSD using which I have >> > created this RAID 0 Disk, wh

[ceph-users] Re: Open discussing: Designing 50GB/s CephFS or S3 ceph cluster

2021-10-21 Thread Christian Wuerdig
- What is the expected file/object size distribution and count? - Is it write-once or modify-often data? - What's your overall required storage capacity? - 18 OSDs per WAL/DB drive seems a lot - recommended is ~6-8 - With 12TB OSD the recommended WAL/DB size is 120-480GB (1-4%) per

[ceph-users] Re: Metrics for object sizes

2021-10-14 Thread Christian Rohmann
bucket stats --bucket mybucket Doing a bucket_size / number_of_objects gives you an average object size per bucket and that certainly is an indication on buckets with rather small objects. Regards Christian ___ ceph-users mailing list -- ceph-users

[ceph-users] Re: CEPH 16.2.x: disappointing I/O performance

2021-10-06 Thread Christian Wuerdig
n a > replicated pool writes and reads are handled by the primary PG, which would > explain this write bandwidth limit. > > /Z > > On Tue, 5 Oct 2021, 22:31 Christian Wuerdig, > wrote: > >> Maybe some info is missing but 7k write IOPs at 4k block size seem f

[ceph-users] Re: CEPH 16.2.x: disappointing I/O performance

2021-10-05 Thread Christian Wuerdig
Maybe some info is missing but 7k write IOPs at 4k block size seem fairly decent (as you also state) - the bandwidth automatically follows from that so not sure what you're expecting? I am a bit puzzled though - by my math 7k IOPS at 4k should only be 27MiB/sec - not sure how the 120MiB/sec was

[ceph-users] Re: Erasure coded pool chunk count k

2021-10-05 Thread Christian Wuerdig
A couple of notes to this: Ideally you should have at least 2 more failure domains than your base resilience (K+M for EC or size=N for replicated) - reasoning: Maintenance needs to be performed so chances are every now and then you take a host down for a few hours or possibly days to do some

  1   2   >