[ceph-users] Re: ceph IO are interrupted when OSD goes down

Octopus 15.2.14? I have totally the same issue and it makes me prod issue. Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com -

[ceph-users] Limit scrub impact

Hi, During scrub I see slow ops like this: osd.31 [WRN] slow request osd_op(client.115442393.0:263257613728.76s0 28:6ed54dc8:::9213182a-14ba-48ad-bde9-289a1c0c0de8.6034919.1_%2fWHITELABEL-1%2fPAGETPYE-7%2fDEVICE-4%2fLANGUAGE-46%2fSUBTYPE-0%2f492210:head [create,setxattr user.rgw.idtag (57) in=

[ceph-users] Re: Metrics for object sizes

attachment ! On 23/04/2021 03:53, Szabo, Istvan (Agoda) wrote: > Objects inside RGW buckets like in couch base software they have their own > metrics and has this information. Not as detailed as you would like, but how about using the bucket stats on bucket size and number of objects?

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

--- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com> --- From: Igor Fedotov Sent: Tuesday, October 12, 2021 7:15 PM To: Szabo, Istvan (Agoda) Cc: ceph-users@ceph.io; 胡 玮文 Subject: Re: [ceph-users]

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

...@agoda.com<mailto:istvan.sz...@agoda.com> --- From: Szabo, Istvan (Agoda) Sent: Tuesday, October 12, 2021 6:54 PM To: Igor Fedotov Cc: ceph-users@ceph.io; 胡 玮文 Subject: RE: [ceph-users] Re: is it possible to remove the db+wal from an external device

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

...@agoda.com<mailto:istvan.sz...@agoda.com> --- From: Igor Fedotov Sent: Tuesday, October 12, 2021 6:45 PM To: Szabo, Istvan (Agoda) Cc: ceph-users@ceph.io; 胡 玮文 Subject: Re: [ceph-users] Re: is it possible to remove the db+wal from an ex

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

re... Thanks, Igor On 10/12/2021 12:13 PM, Szabo, Istvan (Agoda) wrote: Hi Igor, I’ve attached here, thank you in advance. Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.co

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

: Igor Fedotov Sent: Monday, October 11, 2021 10:40 PM To: Szabo, Istvan (Agoda) Cc: ceph-users@ceph.io; Eugen Block ; 胡 玮文 Subject: Re: [ceph-users] Re: is it possible to remove the db+wal from an external device (nvme) Email received from the internet. If in doubt, don't click any link no

[ceph-users] Re: Metrics for object sizes

...@agoda.com<mailto:istvan.sz...@agoda.com> --- From: Yuval Lifshitz Sent: Tuesday, September 14, 2021 7:38 PM To: Szabo, Istvan (Agoda) Cc: Wido den Hollander ; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Metrics for object sizes Email receive

[ceph-users] Re: Where is my free space?

I see, I'm using ssds so it shouldn't be a problem I guess, because the : "bluestore_min_alloc_size": "0" is overwritten with the: "bluestore_min_alloc_size_ssd": "4096" ? -Original Message- From: Stefan Kooman Sent: Tuesday,

[ceph-users] get_health_metrics reporting slow ops and gw outage

Hi, Many of my osds having this issue which causes 10-15ms osd write operation latency and more than 60ms read operation latency. This causes rgw wait for operations and after a while rgw just restarted (all of them in my cluster) and only available after slow ops disappeared. I see similar iss

[ceph-users] Where is my free space?

Hi, 377TiB is the total cluster size, data pool 4:2 ec, stored 66TiB, how can be the data pool on 60% used??!! Some output: ceph df detail --- RAW STORAGE --- CLASS SIZE AVAILUSED RAW USED %RAW USED nvme12 TiB 11 TiB 128 MiB 1.2 TiB 9.81 ssd377 TiB 269 TiB 100

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

com> --- From: Igor Fedotov Sent: Tuesday, October 5, 2021 10:02 PM To: Szabo, Istvan (Agoda) ; 胡 玮文 Cc: ceph-users@ceph.io; Eugen Block Subject: Re: [ceph-users] Re: is it possible to remove the db+wal from an external device (nvme) Email received fr

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

stvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com> ----------- On 2021. Oct 5., at 17:19, Szabo, Istvan (Agoda) wrote:  Hmm, I’ve removed from the cluster, now data rebalance, I’ll do with the next one ☹ Istvan Szabo Senior Infrastructure Engineer

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

com> --- From: Igor Fedotov Sent: Tuesday, October 5, 2021 10:02 PM To: Szabo, Istvan (Agoda) ; 胡 玮文 Cc: ceph-users@ceph.io; Eugen Block Subject: Re: [ceph-users] Re: is it possible to remove the db+wal from an external device (nvme) Email received from the internet. If in

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

--- -Original Message- From: Eugen Block Sent: Tuesday, October 5, 2021 9:52 PM To: Szabo, Istvan (Agoda) Cc: 胡 玮文 ; Igor Fedotov ; ceph-users@ceph.io Subject: Re: [ceph-users] Re: is it possible to remove the db+wal from an external device (nvme) Email received from the

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

": "ceph-osd", "stack_sig": "6a43b6c219adac393b239fbea4a53ff87c4185bcd213724f0d721b452b81ddbf", "timestamp": "2021-10-05T13:31:28.513463Z", "utsname_hostname": "server-2s07", "utsname_machine": &qu

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

com> --- From: Igor Fedotov Sent: Saturday, October 2, 2021 5:22 AM To: Szabo, Istvan (Agoda) Cc: ceph-users@ceph.io; Eugen Block ; Christian Wuerdig Subject: Re: [ceph-users] Re: is it possible to remove the db+wal from an external device (nvme) Email received from the internet.

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

x27;t click any link nor open any attachment ! Hi Istvan, yeah both db and wal to slow migration are supported. And spillover state isn't a show stopper for that. On 10/2/2021 1:16 AM, Szabo, Istvan (Agoda) wrote: Dear Igor, Is the ceph-volume lvm migra

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

com> --- On 2021. Oct 1., at 11:47, Szabo, Istvan (Agoda) wrote: 3x SSD osd /nvme Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- -Or

[ceph-users] Re: Multisite reshard stale instances

enior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- -Original Message- From: Christian Rohmann Sent: Friday, October 1, 2021 10:14 PM To: Szabo, Istvan (Agoda)

[ceph-users] Re: Multisite reshard stale instances

a.com --- -Original Message- From: Christian Rohmann Sent: Friday, October 1, 2021 9:37 PM To: Szabo, Istvan (Agoda) ; ceph-users@ceph.io Subject: Re: [ceph-users] Multisite reshard stale instances Email received from the internet. If in doubt, don&#x

[ceph-users] Re: dealing with unfound pg in 4:2 ec pool

pected? Zitat von "Szabo, Istvan (Agoda)" : > Hi, > > If I set the min size of the pool to 4, will this pg be recovered? > Or how I can take out the cluster from health error like this? > Mark as lost seems risky based on some maillist experience, even if > marked los

[ceph-users] Re: dealing with unfound pg in 4:2 ec pool

enior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- -Original Message- From: Szabo, Istvan (Agoda) Sent: Friday, October 1, 2021 2:50 PM To: Eugen Block ; ceph-

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

o you have? On 10/1/2021 9:55 AM, Szabo, Istvan (Agoda) wrote: > I have my dashboards and I can see that the db nvmes are always running on > 100% utilization (you can monitor with iostat -x 1) and it generates all the > time iowaits which is between 1-3. > > I’m using nvme

[ceph-users] Re: dealing with unfound pg in 4:2 ec pool

keep in mind that if you decrease min_size and you lose another OSD you could face data loss. Are your OSDs still crashing unexpected? Zitat von "Szabo, Istvan (Agoda)" : > Hi, > > If I set the min size of the pool to 4, will this pg be recovered? > Or how I can take o

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

--- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com> --- From: Victor Hooi Sent: Friday, October 1, 2021 5:30 AM To: Eugen Block Cc: Szabo, Istvan (Agoda) ; 胡 玮文 ; ceph-users Subje

[ceph-users] dealing with unfound pg in 4:2 ec pool

Hi, If I set the min size of the pool to 4, will this pg be recovered? Or how I can take out the cluster from health error like this? Mark as lost seems risky based on some maillist experience, even if marked lost after you still have issue, so curious what is the way to take the cluster out fr

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

ock Sent: Thursday, September 30, 2021 1:10 PM To: Szabo, Istvan (Agoda) Cc: 胡 玮文 ; Igor Fedotov ; ceph-users@ceph.io Subject: Re: is it possible to remove the db+wal from an external device (nvme) Email received from the internet. If in doubt, don't cli

[ceph-users] Re: osd_memory_target=level0 ?

5.2.14 becauase that one has buffered_io enabled by default. ty From: Frédéric Nass Sent: Thursday, September 30, 2021 4:43 PM To: Szabo, Istvan (Agoda) ; Christian Wuerdig Cc: Ceph Users Subject: Re: [ceph-users] Re: osd_memory_target=level0 ? Email received from the internet

[ceph-users] Re: osd_memory_target=level0 ?

94 MiB/s wr, 17.70k op/s rd, 2.75k op/s wr recovery: 16 MiB/s, 223 objects/s Ty -Original Message- From: Christian Wuerdig Sent: Thursday, September 30, 2021 1:01 PM To: Szabo, Istvan (Agoda) Cc: Ceph Users Subject: Re: [ceph-users] osd_memory_target=level0 ? Email received from

[ceph-users] osd_memory_target=level0 ?

Hi, Still suffering with the spilledover disks and stability issue in 3 of my cluster after uploaded 6-900 millions objects to the cluster. (Octopus 15.2.10). I’ve set memory target around 31-32GB so could that be that the spilledover issue is coming from here? So have mem target 31GB, next le

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

a.com --- -Original Message- From: Eugen Block Sent: Wednesday, September 29, 2021 8:49 PM To: 胡 玮文 Cc: Igor Fedotov ; Szabo, Istvan (Agoda) ; ceph-users@ceph.io Subject: Re: is it possible to remove the db+wal from an external device (nvme) Email received from the internet. If in

[ceph-users] Re: Billions of objects upload with bluefs spillover cause osds down?

rastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- -Original Message- From: Janne Johansson Sent: Tuesday, September 28, 2021 1:36 PM To: Szabo, Istvan (Agoda) Cc: Ceph Users Subject: Re: [ceph-users]

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

_ Hi, I think 'ceph-bluestore-tool bluefs-bdev-migrate' could be of use here. I haven't tried it in a production environment yet, only in virtual labs. Regards, Eugen Zitat von "Szabo, Istvan (Agoda)" : > Hi, > > Seems like in our config the nvme dev

[ceph-users] Re: Billions of objects upload with bluefs spillover cause osds down?

Regarding point 2, how can it spillover if I wouldn’t use db device just block. Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com

[ceph-users] Billions of objects upload with bluefs spillover cause osds down?

Hi, One of our user is migrating 1.2 billions of objects to one bucket from another system (cassandra) and we are facing in our clusters bluefs spillovers on 50% of the osds. We have 600-900GB dbs bit seems like can’t fit. Also the cluster is very unstable, I can’t really set recovery operation

[ceph-users] is it possible to remove the db+wal from an external device (nvme)

Hi, Seems like in our config the nvme device as a wal+db in front of the ssd slowing down the ssds osds. I'd like to avoid to rebuild all the osd-, is there a way somehow migrate to the "slower device" the wal+db without reinstall? Ty ___ ceph-users

[ceph-users] Re: How you loadbalance your rgw endpoints?

Karlsson Sent: Monday, September 27, 2021 5:44 PM To: Szabo, Istvan (Agoda) Cc: Ceph Users Subject: Re: [ceph-users] How you loadbalance your rgw endpoints? Email received from the internet. If in doubt, don't click any link nor open any attachment ! Hi

[ceph-users] How you loadbalance your rgw endpoints?

Hi, Wonder how you guys do it due to we will always have limitation on the network bandwidth of the loadbalancer. Or if no balancer what to monitor if 1 rgw maxed out? I’m using 15rgw. Ty ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscri

[ceph-users] Re: Modify pgp number after pg_num increased

asted your max-backfill config and it was the lowest possible value (1), right? That's why your backfill is slow. Zitat von "Szabo, Istvan (Agoda)" : > Hi, > > By default in the newer versions of ceph when you increase the pg_num > the cluster will start to increase the

[ceph-users] Modify pgp number after pg_num increased

Hi, By default in the newer versions of ceph when you increase the pg_num the cluster will start to increase the pgp_num slowly up to the value of the pg_num. I've increased the ec-code data pool from 32 to 128 but 1 node has been added to the cluster and it's very slow. pool 28 'hkg.rgw.bucket

[ceph-users] Re: RocksDB options for HDD, SSD, NVME Mixed productions

ervices Co., Ltd. e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com> --- On 2021. Sep 21., at 20:21, Christian Wuerdig wrote: Email received from the internet. If in doubt, don't click any link nor open any attachment ! _

[ceph-users] Re: RocksDB options for HDD, SSD, NVME Mixed productions

Sorry to steal it, so if I have 500GB and 700GB mixed wal+rocksdb on nvme the number should be the level base 50 and 70? Or needs to be power of 2? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.

[ceph-users] Re: RocksDB options for HDD, SSD, NVME Mixed productions

Sorry to steal it, so if I have 500GB and 700GB mixed wal+rocksdb on nvme the number should be the level base 50 and 70? Or needs to be power of 2? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.s

[ceph-users] Re: RocksDB options for HDD, SSD, NVME Mixed productions

Let me join, having 11 bluefs spillover in my cluster. Where this settings coming from? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com ---

[ceph-users] Safe value for maximum speed backfilling

Hi, 7 node, ec 4:2 host based crush, ssd osds with nvme wal+db, what shouldn't cause any issue with these values? osd_max_backfills = 1 osd_recovery_max_active = 1 osd_recovery_op_priority = 1 I want to speed it up but haven't really found any reference. Ty

[ceph-users] Re: Adding cache tier to an existing objectstore cluster possible?

ineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- -Original Message- From: Stefan Kooman Sent: Monday, September 20, 2021 2:13 PM To: Szabo, Istvan (Agoda) ; ceph-users Subject: Re: [ceph-

[ceph-users] Adding cache tier to an existing objectstore cluster possible?

Hi, I'm running out of idea why my wal+db nvmes are maxed out always so thinking of I might missed the cache tiering in front of my 4:2 ec-pool. IS it possible to add it later? There are 9 nodes with 6x 15.3TB SAS ssds, 3x nvme drives. Currently out of the 3 nvme 1 is used for index pool and me

[ceph-users] Buffered io +/vs osd memory target

Hi, If we are using buffered_io does the osd memory target settings still makes any sense (or vice-versa)? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com ---

[ceph-users] Re: BLUEFS_SPILLOVER

-Original Message- From: Janne Johansson Sent: Thursday, September 16, 2021 1:03 PM To: Szabo, Istvan (Agoda) Cc: ceph-users Subject: Re: [ceph-users] BLUEFS_SPILLOVER Email received from the internet. If in doubt, don't click any link nor open any attachment ! __

[ceph-users] Endpoints part of the zonegroup configuration

Hi, In the documentation is not really clear the endpoints under the zone and under the zonegroup "collection" part. 1. If you have a loadbalancer in front of the gateways, should you put the lb in these sections or always put the individual gateway list? Having this configuration: jpst.it/2C

[ceph-users] BLUEFS_SPILLOVER

Hi, Something weird happening, I have on 1 nvme drive and 3x SSD's are using for wal and db. The LVM is 596GB but in the health detail is says x GiB spilled over to slow device, however just 317 GB use only :/ [WRN] BLUEFS_SPILLOVER: 3 OSD(s) experiencing BlueFS spillover osd.10 spilled ov

[ceph-users] Re: OSD based ec-code

" like 12:3 or ? > > You should evaluate that the other way around. What are your specific > requirements regarding resiliency (how many hosts can fail at the same > time without data loss)? How many hosts are available? Are you > planning to expand in the near future? Based

[ceph-users] Re: Metrics for object sizes

at 4:53 AM Szabo, Istvan (Agoda) mailto:istvan.sz...@agoda.com>> wrote: Objects inside RGW buckets like in couch base software they have their own metrics and has this information. Istvan Szabo Senior Infrastructure Engineer --- Agoda Ser

[ceph-users] Re: How many concurrent users can be supported by a single Rados gateway

m wondering what is the major benifit of running multiple RGWs instead of a single one on baremetal machine? Is it because that a single RGW has some inherent limitations on using multi threads? thanks, samuel huxia...@horebdata.cn From: Szabo, Is

[ceph-users] Re: [Suspicious newsletter] Problem with multi zonegroup configuration

I don't see any sync rule like you want to do directional sync between 2 zones, no pipe and no flow also. Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com

[ceph-users] data rebalance super slow

Hi, I have a 4:2 erasure coded data pool which based on host and have 6 data nodes. I've increased the pg from 32 to 128 but the recovery operation is less than 10MB/s. The server has bonded 25GB nic cards. In octopus I don't have to initiate the pgp_increase because it automatically adjust up,

[ceph-users] Re: How many concurrent users can be supported by a single Rados gateway

com> --- From: huxia...@horebdata.cn Sent: Monday, September 13, 2021 2:24 PM To: Szabo, Istvan (Agoda) ; Eugen Block Cc: ceph-users Subject: Re: RE: [ceph-users] Re: How many concurrent users can be supported by a single Rados gateway Email received from the internet. If in doubt, don't cli

[ceph-users] Ceph advisor for objectstore

Hi, Our cluster is getting more critical and I might want to revisit the design deeply from even the os tuning part so curious is there any company who is known as an objectstore expert? Thank you. ___ ceph-users mailing list -- ceph-users@ceph.io To

[ceph-users] OSD based ec-code

Hi, What's your take on an osd based ec-code setup? I've never been brave enough to use OSD based crush rule because scared host failure but in the last 4 years we have never had any host issue so I'm thinking to change to there and use some more cost effective EC. Also what is the "optimal" l

[ceph-users] Re: How many concurrent users can be supported by a single Rados gateway

Good topic, I'd be interested also. One of the redhat document says 1GW / 50 OSD, but I think it is not a relevant formula. I had couple of time when the users doing something stupid and totally ddos down the hole cluster. What I've done added additional 4 rgw in each of the mon/mgr nodes where

[ceph-users] Bluefs spillover octopus 15.2.10

Hi, I’ve looked around what to do with this but haven’t really found any solution, so I wonder if I have this alert: osd.32 spilled over 324 MiB metadata from 'db' device (317 GiB used of 596 GiB) to slow device What can I actually do? Many thread relates to what developers will do :/ Thank y

[ceph-users] Is autoscale working with ec pool?

Hi, In our cluster we only the data pool is on ec 4:2, the others are on replica 3. --- RAW STORAGE --- CLASS SIZE AVAILUSED RAW USED %RAW USED nvme10 TiB 10 TiB 121 MiB 408 GiB 3.80 ssd524 TiB 440 TiB 84 TiB84 TiB 16.01 TOTAL 534 TiB 450 TiB 84 Ti

[ceph-users] pg_num number for an ec pool

Hi, For replicated pool the calculation is easy but I might be confused with replicated. I have 36 osd (15TB each), how much pg should be the data pool for objectstore located on ec 4:2? Am I correct with 512? So let's say 100pg/osd, so 36x100/6 and closest power of 2? Thank you.

[ceph-users] Re: LARGE_OMAP_OBJECTS: any proper action possible?

Can I steal this thread just for 1 answer 😊 ? I have 11 omap in my octopus cluster related to datalog like this: /var/log/ceph/ceph.log-20210822.gz:2021-08-21T09:06:20.605200+0700 osd.11 (osd.11) 1876 : cluster [WRN] Large omap object found. Object: 22:b040fc05:::data_log.31:head PG: 22.a03f020

[ceph-users] Re: [Suspicious newsletter] Re: s3 select api

a.com --- -Original Message- From: Konstantin Shalygin [mailto:k0...@k0ste.ru] Sent: Wednesday, September 1, 2021 1:38 PM To: Szabo, Istvan (Agoda) Cc: Ceph Users Subject: [Suspicious newsletter] [ceph-users] Re: s3 select api Email received from the internet. If in doubt, don't clic

[ceph-users] s3 select api

Hi, Is the s3 select api working with octopus or only with pacific? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Adding a new monitor causes cluster freeze

Any reason to use kernel 5 rather than 3? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- On 2021. Aug 30., at 10:2

[ceph-users] data_log omaps

Hi, I have 11 omap in my octopus cluster related to datalog like this: /var/log/ceph/ceph.log-20210822.gz:2021-08-21T09:06:20.605200+0700 osd.11 (osd.11) 1876 : cluster [WRN] Large omap object found. Object: 22:b040fc05:::data_log.31:head PG: 22.a03f020d (22.d) Key count: 436895 Size (bytes):

[ceph-users] Re: Max object size GB or TB in a bucket

--- -Original Message- From: Janne Johansson Sent: Friday, August 20, 2021 3:52 PM To: Marc Cc: Szabo, Istvan (Agoda) ; Ceph Users Subject: Re: [ceph-users] Re: Max object size GB or TB in a bucket Email received from the internet. If in doubt, don't click any link nor open

[ceph-users] Re: Max object size GB or TB in a bucket

, August 20, 2021 3:16 PM To: Szabo, Istvan (Agoda) Cc: Ceph Users Subject: Re: [ceph-users] Max object size GB or TB in a bucket Email received from the internet. If in doubt, don't click any link nor open any attachment ! Den fre 20 aug. 2021 kl 09:20

[ceph-users] Re: Max object size GB or TB in a bucket

21 kl 08:32 skrev Szabo, Istvan (Agoda) : These are the values in octopus: "rgw_max_put_size": "5_368_709_120", "rgw_multipart_part_upload_limit": "1", "rgw_multipart_min_part_size": "5242880", Correct me if I'm wrong but

[ceph-users] Max object size GB or TB in a bucket

Hi, These are the values in octopus: "rgw_max_put_size": "5368709120", "rgw_multipart_part_upload_limit": "1", "rgw_multipart_min_part_size": "5242880", Correct me if I'm wrong but the multipart parts size is 15MB so 1 means maximum size of one object is 150GB. What is the h

[ceph-users] Re: [Suspicious newsletter] Re: create a Multi-zone-group sync setup

Hi, " but have a global namespace where all buckets and users are uniqe." You mean manage multiple cluster from 1 "master" cluster but ono sync? So 1 realm, multiple dc BUT no sync? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co

[ceph-users] Re: [Suspicious newsletter] Ceph cluster with 2 replicas

Hi, We are using with replica 2 also but we have copy of the master data, just be careful with replica 2. Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com ---

[ceph-users] Re: [Suspicious newsletter] Re: RGW memory consumption

d memory usage is gone. I will be watching all Rgw's to understand beter. I give +1 vote, its probably memory leak. 14 Ağu 2021 Cmt 09:56 tarihinde Szabo, Istvan (Agoda) mailto:istvan.sz...@agoda.com>> şunu yazdı: Are you using loadbalancer? Maybe you use source based balancing method

[ceph-users] Re: [Suspicious newsletter] Re: RGW memory consumption

Are you using loadbalancer? Maybe you use source based balancing method? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com ---

[ceph-users] Re: [Suspicious newsletter] Re: [Suspicious newsletter] Bucket creation on RGW Multisite env.

com> --- On 2021. Aug 5., at 11:13, Soumya Koduri wrote:  On 8/5/21 12:55 PM, Szabo, Istvan (Agoda) wrote: Hi, I’m using 15.2.10. Global sync you mean complete site mirror is working only? We’ve tried to made it work like half year or even more, the g

[ceph-users] Re: [Suspicious newsletter] Re: [Suspicious newsletter] Bucket creation on RGW Multisite env.

Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com> --- On 2021. Aug 4., at 17:40, Soumya Koduri wrote: On 8/4/21 10:12 AM, Szabo, Istvan

[ceph-users] Re: [Suspicious newsletter] Bucket creation on RGW Multisite env.

---- From: Soumya Koduri Sent: Tuesday, June 8, 2021 7:05 PM To: Szabo, Istvan (Agoda) ; mhnx Cc: Ceph Users Subject: Re: [ceph-users] Re: [Suspicious newsletter] Bucket creation on RGW Multisite env. On 6/8/21 4:59 PM, Szabo, Istvan (Agoda) wrote: > Yes, but with this the bucket contents wil

[ceph-users] Octopus in centos 7 with kernel 3

Hi, I saw couple of discussions in the mail list about this topic, so is it working properly or not? Ceph documentation says octopus needs kernel 4. Thank you This message is confidential and is for the sole use of the intended recipient(s). It may also be priv

[ceph-users] How to set retention on a bucket?

Hi, Haven't really found how to set the retention on a s3 bucket for a specific day. Is there any ceph document about it? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com

[ceph-users] Re: Installing and Configuring RGW to an existing cluster

You have different ways: ceph-deploy and full manual: Full manual: RGW: on all RGW yum install ceph-radosgw -y first RGW node: ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring chown ceph:ceph /etc/ceph/ceph.client.radosgw.keyring ceph-authtool /etc/ceph/ceph.client.radosgw.keyr

[ceph-users] Re: How to size nvme or optane for index pool?

.sz...@agoda.com> --- On 2021. Jul 15., at 13:24, Konstantin Shalygin wrote: What you mean? You can check pool usage via 'ceph df detail' output Sent from my iPhone On 15 Jul 2021, at 07:53, Szabo, Istvan (Agoda) wrote: How can I know w

[ceph-users] Re: How to size nvme or optane for index pool?

nd speed up recovery. On Jul 14, 2021, at 9:52 PM, Szabo, Istvan (Agoda) wrote: Hi, How can I know which size of the nvme drive needed for my index pool? At the moment I'm using 6x1.92TB NVME (overkill) but I have no idea how is it used. Thanks

[ceph-users] How to size nvme or optane for index pool?

Hi, How can I know which size of the nvme drive needed for my index pool? At the moment I'm using 6x1.92TB NVME (overkill) but I have no idea how is it used. Thanks This message is confidential and is for the sole use of the intended recipient(s). It may also b

[ceph-users] "missing required protocol features" when map rbd

Hi, We have updated our cluster from luminous 12.2.8 to nautilus 14.2.22 and users with this core kernel: Linux servername 4.9.241-37.el7.x86_64 #1 SMP Mon Nov 2 13:55:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Experiencing this issue when they try to map the image. (unmap works) [15085691.

[ceph-users] Re: [Suspicious newsletter] Issue with Nautilus upgrade from Luminous

I've just made update this week also but mine required jewel at least. Hadn't it notify about that before? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com ---

[ceph-users] Re: NVME hosts added to the clusters and it made old ssd hosts flapping osds

her over it. If you’re using jumbo frames, ensure that the old and new nodes / switches are all configured appropriately. On Jul 8, 2021, at 4:17 AM, Szabo, Istvan (Agoda) wrote: Hi, I've added 4 nvme hosts with 2osd/nvme to my cluster and it made al the ssd osds flapping I don't

[ceph-users] NVME hosts added to the clusters and it made old ssd hosts flapping osds

Hi, I've added 4 nvme hosts with 2osd/nvme to my cluster and it made al the ssd osds flapping I don't understand why. It is under the same root but 2 different device classes, nvme and ssd. The pools are on the ssd on the nvme nothing at the moment. The only way to bring back the ssd osds alive t

[ceph-users] list-type=2 requests

Hi, Is there anybody know about list-type=2 request? GET /bucket?list-type=2&max-keys=2 We faced yesterday the 2nd big objectstore cluster outage due to this request. 1 user made the cluster down totally. The normal ceph iostat read operation is below 30k, when they deployed their release it ju

[ceph-users] Haproxy config, multilple RGW on the same node with different ports haproxy ignore

Hi, I have this config: https://jpst.it/2yBsD What I'm missing from the backend part to make it able to balance on the same server but different port? Thank you This message is confidential and is for the sole use of the intended recipient(s). It may also be

[ceph-users] Objectstore user IO and operations monitoring

Hi, I'm looking for this long time ago, I have a lot of users and when 1 user can take down the cluster I want to know which one, but there isn't any bucket stats that could help. Anyone knows anything? Istvan Szabo Senior Infrastructure Engineer

[ceph-users] Remove objectstore from a RBD RGW cluster

Hi, I want to remove all the objectstore related things from my cluster and keep only for RBD. I've uninstalled the RGW services. Removed the haproxy config related to that. When I try to delete realm, zone, zonegroup it is finished but after coupe of minutes something recreate another zonegro

[ceph-users] Ceph connect to openstack

Hi, Is there any proper documentation how to connect ceph with openstack? This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have receive

[ceph-users] Ceph DB

Hi, If I set in a running cluster the rgw_bucket_default_quota_max_objects with the ceph config set (mgr/mon not sure which to make it global) rgw_bucket_default_quota_max_objects 100 will it overwrite the existing special buckets where I've already set higher values? Or keep in untouched?

[ceph-users] Re: [Suspicious newsletter] Nic bonding (lacp) settings for ceph

We are using it with 3+4. Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- -Original Message- From: mhnx Sent: Saturday, June 26, 2021 8:4

[ceph-users] Re: [Suspicious newsletter] In theory - would 'cephfs root' out-perform 'rbd root'?

Not really clear for me to be ho mnest how many cephfs is needed? Would it worth to create multiple or what is the use case to create multiple? In the examples and how people using it seems like only 1 cephfs + 1 metadata pool on nvme, not really multiple cephfs. Doc just relates that if you wa

[ceph-users] Re: CephFS design

tioned that "main use case would be k8s users", are there 2000 users that need 500 IOPS each at 100MB/s, OR 5 users that touch the storage every few minutes and store 10MB of data. These are polar opposites with orders of magnitude different requirements. On Fri, Jun 11, 2021 at 10:56

<    1   2   3   4   >