[ceph-users] Re: All monitors failed, recovering from encrypted osds: everything lost??

Den ons 25 aug. 2021 kl 14:27 skrev Ignacio García : > > Only 1 monitor that was running on a failed disk -> unrecoverable > store.db to create a new monitor. > > Then trying to recover from osds following: > https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#recovery-using-

[ceph-users] Re: Pacific: access via S3 / Object gateway slow for small files

Den tis 24 aug. 2021 kl 09:46 skrev Francesco Piraneo G. : > Il 24.08.21 09:32, Janne Johansson ha scritto: > >> As a simple test I copied an Ubuntu /usr/share/doc (580 MB in 23'000 > >> files): > >> - rsync -a to a Cephfs took 2 min > >> - s3cmd

[ceph-users] Re: Pacific: access via S3 / Object gateway slow for small files

Den tis 24 aug. 2021 kl 09:12 skrev E Taka <0eta...@gmail.com>: > As a simple test I copied an Ubuntu /usr/share/doc (580 MB in 23'000 files): > > - rsync -a to a Cephfs took 2 min > - s3cmd put --recursive took over 70 min > Users reported that the S3 access is generally slow, not only with s3tool

[ceph-users] Re: Max object size GB or TB in a bucket

Den fre 20 aug. 2021 kl 10:45 skrev Marc : > > > > S3cmd chunks 15MB. > > There seems to be an s5cmd, which should be much much faster than s3cmd. There is both s4cmd, s5cmd, minio-mc and rclone which all have some things that make them "better" than s3cmd in various ways, at the expense of lackin

[ceph-users] Re: Max object size GB or TB in a bucket

Den fre 20 aug. 2021 kl 10:34 skrev Szabo, Istvan (Agoda) : > > So 1x5GB so like 50TB/file? Yes. Aws seems to have a max of 1000 pieces, which gives them a max size of 5TB, but from your numbers, it seems you have higher limits than aws. -- May the most significant bit of your life be posit

[ceph-users] Re: Max object size GB or TB in a bucket

Den fre 20 aug. 2021 kl 09:20 skrev Szabo, Istvan (Agoda) : > > S3cmd chunks 15MB. No, s3cmd chunks the size you tell it to, but defaults to some value. --multipart-chunk-size-mb=SIZE Size of each chunk of a multipart upload. Files bigger than SIZ

[ceph-users] Re: Max object size GB or TB in a bucket

Den fre 20 aug. 2021 kl 08:32 skrev Szabo, Istvan (Agoda) : > These are the values in octopus: > "rgw_max_put_size": "5_368_709_120", > "rgw_multipart_part_upload_limit": "1", > "rgw_multipart_min_part_size": "5242880", > > Correct me if I'm wrong but the multipart parts size is 15M

[ceph-users] Re: Manual deployment of an OSD failed

Den ons 18 aug. 2021 kl 21:49 skrev Francesco Piraneo G. : > > Il 17.08.21 16:34, Marc ha scritto: > > > ceph-volume lvm zap --destroy /dev/sdb > > ceph-volume lvm create --data /dev/sdb --dmcrypt > > > > systemctl enable ceph-osd@0 > > > Hi Marc, > > it worked! Thank you very much! > > I have some

[ceph-users] Re: Multiple DNS names for RGW?

Den ons 18 aug. 2021 kl 08:41 skrev Christian Rohmann : > > On 17/08/2021 13:37, Janne Johansson wrote: > > Don't forget that v4 auth bakes in the clients idea of what the > > hostname of the endpoint was, so its not only about changing headers. > > If you are not u

[ceph-users] Re: Raid redundance not good

Den tis 17 aug. 2021 kl 12:17 skrev Network Admin : > Hi all, > first, apologize for my english writen :) > I installed a Ceph system with 3 servers : > - server 1 : all services > - server 2 : all services > - serveur 3 : no osd, only monitor > I put files with Cepfs : all is good and ceph monitor

[ceph-users] Re: Multiple DNS names for RGW?

Den tis 17 aug. 2021 kl 11:46 skrev Chris Palmer : > > Hi Christian > > I don't have much experience with multisite so I'll let someone else > answer that aspect. But each RGW will only accept requests where the > Host header matches one of the "hostnames" configured as below. > Otherwise the clien

[ceph-users] Re: Multiple DNS names for RGW?

Den mån 16 aug. 2021 kl 08:53 skrev Burkhard Linke : > Hi, > we are running RGW behind haproxy for TLS termination and load > balancing. Due to some major changes in our setup, we would like to > start a smooth transition to a new hostname of the S3 endpoint. The > haproxy part should be straightfo

[ceph-users] Re: Discard / Trim does not shrink rbd image size when disk is partitioned

Den tors 12 aug. 2021 kl 17:04 skrev Boris Behrens : > Hi everybody, > we just stumbled over a problem where the rbd image does not shrink, when > files are removed. > This only happenes when the rbd image is partitioned. > > * We tested it with centos8/ubuntu20.04 with ext4 and a gpt partition tab

[ceph-users] Re: Is it a bad Idea to build a Ceph Cluster over different Data Centers?

Den tis 10 aug. 2021 kl 21:59 skrev Ralph Soika : > communicate via public IPs. I did some tests and - not surprising - the > network performance is going down to 500Mbit/s between two nodes in two > different data centers. > My questions are: > > 1. Is 500Mbit/s to slow to run a ceph cluster? Th

[ceph-users] Re: we're living in 2005.

Den mån 26 juli 2021 kl 16:56 skrev : > and there's an irc channel that's dead. This might have an explanation, presumably because of tons of spam bots, the ceph channel (or whole oftc network?) requires you to be registered (which I wasn't), so I was posting a lot to IRC these last weeks and felt

[ceph-users] Re: Rogue osd / CephFS / Adding osd

Den fre 30 juli 2021 kl 15:22 skrev Thierry MARTIN : > Hi all ! > We are facing strange behaviors from two clusters we have at work (both > v15.2.9 / CentOS 7.9): > * In the 1st cluster we are getting errors about multiple degraded pgs > and all of them are linked with a "rogue" osd which ID

[ceph-users] Re: we're living in 2005.

Den tis 27 juli 2021 kl 12:32 skrev Rok Jaklič : > Actually, some of us tried to contribute to documentation but were stopped > with failed build checks for some reason. > While most of it is ok, at some places documentation is vague or missing > (maybe also the reason why this thread is so long al

[ceph-users] Re: we're living in 2005.

Den tis 27 juli 2021 kl 10:21 skrev Wido den Hollander : > > Try to install a completely new ceph cluster from scratch on fresh > > installed LTS Ubuntu by this doc > > https://docs.ceph.com/en/latest/cephadm/install/ . Many interesting > > discoveries await you. > > Nothing special - only step

[ceph-users] Re: we're living in 2005.

Den tis 27 juli 2021 kl 10:09 skrev Marc : > > > > Try to install a completely new ceph cluster from scratch on fresh > > installed LTS Ubuntu by this doc > > https://docs.ceph.com/en/latest/cephadm/install/ . Many interesting > > discoveries await you. > > on centos7 14.2.22, manual with no surpri

[ceph-users] Re: we're living in 2005.

Den tis 27 juli 2021 kl 05:12 skrev Fyodor Ustinov : > > Hi! > > > docs.ceph.io ? If there’s something that you’d like to see added there, > > you’re > > welcome to submit a tracker ticket, or write to me privately. It is not > > uncommon for documentation enhancements to be made based on mailin

[ceph-users] Re: Remove objectstore from a RBD RGW cluster

Sounds like an rgw is still running. Den mån 5 juli 2021 08:15Szabo, Istvan (Agoda) skrev: > Hi, > > I want to remove all the objectstore related things from my cluster and > keep only for RBD. > > I've uninstalled the RGW services. > Removed the haproxy config related to that. > > When I try to

[ceph-users] Re: Octopus support

Den tis 22 juni 2021 kl 15:44 skrev Shafiq Momin : > I see octopus is having limited Suport on Centos 7 I have prod cluster with > 1.2 PTB data with nautilus 14.2 > Can we upgrade on Centos 7 from nautilus to octopus or we foresee issue Upgrading to octopus should be fine, we run C7 cluster with t

[ceph-users] Re: HDD <-> OSDs

Den tis 22 juni 2021 kl 11:55 skrev Thomas Roth : > Hi all, > newbie question: > The documentation seems to suggest that with ceph-volume, one OSD is created > for each HDD (cf. 4-HDD-example in > https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/) > > This seems odd: what i

[ceph-users] Re: Upgrading ceph to latest version, skipping minor versions?

Den mån 14 juni 2021 kl 22:48 skrev Matt Larson : > > Looking at the documentation for ( > https://docs.ceph.com/en/latest/cephadm/upgrade/) - I have a question on > whether you need to sequentially upgrade for each minor versions, 15.2.1 -> > 15.2.3 -> ... -> 15.2.XX? > > Can you safely upgrade by

[ceph-users] Re: Integration of openstack to ceph

Have you checked https://docs.ceph.com/en/latest/radosgw/keystone/ ? Den tors 10 juni 2021 kl 10:06 skrev Michel Niyoyita : > > Dear Ceph Users, > > Anyone can help on the guidance of how I can integrate ceph to openstack ? > especially RGW. > > Regards > > Michel > __

[ceph-users] Re: ceph buckets

Den tis 8 juni 2021 kl 14:31 skrev Rok Jaklič : > Which mode is that and where can I set it? > This one described in https://docs.ceph.com/en/latest/radosgw/multitenancy/ ? Yes, the description says it all there, doesn't it? >> >> Apart from that, there is a mode for RGW with tenant/bucketname wh

[ceph-users] Re: ceph buckets

Den tis 8 juni 2021 kl 12:38 skrev Rok Jaklič : > Hi, > I try to create buckets through rgw in following order: > - *bucket1* with *user1* with *access_key1* and *secret_key1* > - *bucket1* with *user2* with *access_key2* and *secret_key2* > > when I try to create a second bucket1 with user2 I get

[ceph-users] Re: OT: How to Build a poor man's storage with ceph

Den tis 8 juni 2021 kl 11:39 skrev Ml Ml : > Maybe combine 3x 10TB HDDs to a 30TB Raid0/striping Disk => which > would speed up the performance, but have a bigger impact on a dying > disk. ^^ This sounds like a very bad idea. When this 30T monster fails, you will have to wait for 30TB to reb

[ceph-users] radosgw-admin bucket delete linear memory growth?

I am seeing a huge usage of ram, while my bucket delete is churning over left-over multiparts, and while I realize there are *many* being done a 1000 at a time, like this: 2021-06-03 07:29:06.408 7f9b7f633240 0 abort_bucket_multiparts WARNING : aborted 254000 incomplete multipart uploads ..my fi

[ceph-users] Re: Can we deprecate FileStore in Quincy?

Den tis 1 juni 2021 kl 21:24 skrev Neha Ojha : > Hello everyone, > > Given that BlueStore has been the default and more widely used > objectstore since quite some time, we would like to understand whether > we can consider deprecating FileStore in our next release, Quincy and > remove it in the R r

[ceph-users] Re: Remapping OSDs under a PG

Create a crush rule that only chooses non-ssd drives, then ceph osd pool set crush_rule YourNewRuleName and it will move over to the non-ssd OSDs. Den fre 28 maj 2021 kl 02:18 skrev Jeremy Hansen : > > > I’m very new to Ceph so if this question makes no sense, I apologize. > Continuing to study

[ceph-users] Re: best practice balance mode in HAproxy in front of RGW?

Den ons 26 maj 2021 kl 16:33 skrev Boris Behrens : > > Hi Janne, > do you know if there can be data duplication which leads to orphan objects? > > I am currently huntin strange errors (there is a lot more data in the > pool, than accessible via rgw) and want to be sure it doesn't come > from the HA

[ceph-users] Re: best practice balance mode in HAproxy in front of RGW?

I guess normal round robin should work out fine too, regardless of if there are few clients making several separate connections or many clients making a few. Den ons 26 maj 2021 kl 12:32 skrev Boris Behrens : > > Hello togehter, > > is there any best practive on the balance mode when I have a HApr

[ceph-users] Re: cephfs vs rbd vs rgw

> So, its not necessarily a "which one should I support". One of cephs great > features is you can support all 3 with the same storage and use them all as > needed. ..with the caveat that you can't serve the same files over them, but it is quite true that you can have all three served from the s

[ceph-users] Re: Very uneven OSD utilization

I would suggest enabling the upmap balancer if you haven't done that, it should help even data out. Even if it would not do better than some manual rebalancing scheme, it will at least do it nicely in the background some 8 PGs at a time so it doesn't impact client traffic. I looks very weird to ha

[ceph-users] Re: OSD and RBD on same node?

Den tis 25 maj 2021 kl 02:51 skrev Tuffli, Chuck : > I found a warning in Red Hat documentation regarding OSD and RBD: > Ceph Block Devices must be deployed on separate nodes from the Ceph Monitor > and OSD nodes. Running kernel clients and kernel server daemons on the same > node can lead to ker

[ceph-users] Re: How to organize data in S3

Den mån 24 maj 2021 kl 07:51 skrev Michal Strnad : > > Thank you. So we can create millions of buckets associated to only one > S3 account without any limitation or side effect? Does anyone use it > this way? I'm not sure listing buckets will be fun if you literally create "millions of buckets" e

[ceph-users] Re: How to organize data in S3

Many buckets. Den sön 23 maj 2021 kl 20:53 skrev Michal Strnad : > > Hi all, > > We need to store millions of files using S3 protocol in Ceph (version > Nautilus), but have projects where isn't appropriate or possible to > create a lot of S3 accounts. Is it better to have multiple S3 buckets or >

[ceph-users] Re: ceph osd df size shows wrong, smaller number

Den fre 21 maj 2021 kl 10:49 skrev Rok Jaklič : > It shows > sdb8:16 0 5.5T 0 disk /var/lib/ceph/osd/ceph-56 That one says osd-56, you asked about why osd 85 was small in ceph osd df >> Den fre 21 maj 2021 kl 09:41 skrev Rok Jaklič : >> > why would ceph osd df show in SI

[ceph-users] Re: ceph osd df size shows wrong, smaller number

Den fre 21 maj 2021 kl 09:41 skrev Rok Jaklič : > why would ceph osd df show in SIZE field smaller number than there is: > 85hdd 0.8 1.0 100 GiB 96 GiB 95 GiB 289 KiB 952 > MiB 4.3 GiB 95.68 3.37 10 up > > instead of 100GiB there should be 5.5TiB. What does "lsbl

[ceph-users] Re: [Suspicious newsletter] Re: bluefs_buffered_io turn to true

Den mån 17 maj 2021 kl 08:15 skrev Szabo, Istvan (Agoda) < istvan.sz...@agoda.com>: > What happens if we are using buffered_io and the machine restared due to > some power failure? Everything that was in the cache will be lost or how > ceph handle this? > Not to be picky, but between any client w

[ceph-users] Re: pgremapper released

Looks great! Den ons 5 maj 2021 kl 15:27 skrev Josh Baergen : > > Hello all, > > I just wanted to let you know that DigitalOcean has open-sourced a > tool we've developed called pgremapper. > > Originally inspired by CERN's upmap exception table manipulation > scripts, pgremapper is a CLI written

[ceph-users] Re: 14.2.20: Strange monitor problem eating 100% CPU

Den tis 4 maj 2021 kl 16:29 skrev Dan van der Ster : > BTW, if you find that this is indeed what's blocking your mons, you > can workaround by setting `ceph progress off` until the fixes are > released. Most ceph commands (and a few of the ceph daemon commands) would just block, so I guess one wou

[ceph-users] Re: 14.2.20: Strange monitor problem eating 100% CPU

Den tis 4 maj 2021 kl 16:10 skrev Rainer Krienke : > Hello, > I am playing around with a test ceph 14.2.20 cluster. The cluster > consists of 4 VMs, each VM has 2 OSDs. The first three VMs vceph1, > vceph2 and vceph3 are monitors. vceph1 is also mgr. > What I did was quite simple. The cluster is in

[ceph-users] Re: Different ceph versions on nodes in cluster

Den fre 23 apr. 2021 kl 08:45 skrev Eugen Block : > you can't really avoid having different versions in a cluster, think > about an update/upgrade. Usually it's not an issue, but I would > recommend to have that state for as short as possible. Just a recently > a customer complained about inconsist

[ceph-users] Re: Upmap balancer after node failure

Den fre 2 apr. 2021 kl 11:23 skrev Dan van der Ster : > > Hi again, > > Oops, I'd missed the part about some PGs being degraded, which > prevents the balancer from continuing. > any upmaps which are directing PGs *to* those toofull OSDs. Or maybe > it will be enough to just reweight those OSDs to 0

[ceph-users] Re: memory consumption by osd

Den sön 28 mars 2021 kl 04:59 skrev Tony Liu : > > I don't see any problems yet. All OSDs are working fine. > Just that 1.8GB free memory concerns me. Don't be concerned. The real amount you should look for is free+buffer+cache. If that is large, then you have no problems. This is a very old conce

[ceph-users] Re: add and start OSD without rebalancing

Den ons 24 mars 2021 kl 14:55 skrev Boris Behrens : > > Oh cool. Thanks :) > > How do I find the correct weight after it is added? > For the current process I just check the other OSDs but this might be a > question that someone will raise. > > I could imagine that I need to adjust the ceph-gentle

[ceph-users] Re: add and start OSD without rebalancing

Den ons 24 mars 2021 kl 14:27 skrev Dan van der Ster : > You can use: >osd_crush_initial_weight = 0.0 We have it at 0.001 or something low which is non-zero so it doesn't start as "out" or anything, but still will not receive any PGs. -- May the most significant bit of your life be positive.

[ceph-users] Re: ceph-ansible in Pacific and beyond?

Den ons 17 mars 2021 kl 20:17 skrev Matthew H : > > "A containerized environment just makes troubleshooting more difficult, > getting access and retrieving details on Ceph processes isn't as > straightforward as with a non containerized infrastructure. I am still not > convinced that containeriz

[ceph-users] Re: Networking Idea/Question

Den ons 17 mars 2021 kl 02:04 skrev Tony Liu : > What's the purpose of "cluster" network, simply increasing total > bandwidth or for some isolations? Not having client traffic (that only occurs on the public network) fight over bandwidth with OSD<->OSD traffic (replication and recovery). Now, th

[ceph-users] Re: Ceph Cluster Taking An Awful Long Time To Rebalance

> pgs: 88.889% pgs not active > 6/21 objects misplaced (28.571%) > 256 creating+incomplete For new clusters, "creating+incomplete" sounds like you created a pool (with 256 PGs) with some crush rule that doesn't allow it to find suitable placements, like "replic

[ceph-users] Re: How big an OSD disk could be?

Den lör 13 mars 2021 kl 12:56 skrev Marc : > > A good mix of size and performance is the Seagate 2X14 MACH.2 Dual > > Actor 14TB HDD. > > This drive reports as 2x 7TB individual block devices and you install > > a OSD on each. > > My first thought was, wow quite nice this dual exposes itself as two

[ceph-users] Re: How big an OSD disk could be?

Den fre 12 mars 2021 kl 18:10 skrev huxia...@horebdata.cn : > Dear cephers, > Just wonder how big an OSD disk could be? Currently the biggest HDD has a > capacity of 18TB or 20TB. It is suitable for an OSD still? > Is there a limitation of the capacity of a single OSD? Can it be 30TB , 50TB > or

[ceph-users] Re: cephadm (curl master)/15.2.9:: how to add orchestration

Den tors 11 mars 2021 kl 13:56 skrev Adrian Sevcenco : > apparently i lack orchestration .. the the documentation show a big > "Page does not exist" > see > https://docs.ceph.com/en/latest/docs/octopus/mgr/orchestrator > Where does this link come from? Usually "latest" and an actual release name

[ceph-users] Re: ceph-radosgw: Initialization timeout, failed to initialize

Den tis 23 feb. 2021 kl 16:53 skrev Mathew Snyder : > > We have a Red Hat installation of Luminuous (full packages version: > 12.2.8-128.1). We're experiencing an issue where the ceph-radosgw service > will timeout during initialization and cycle through attempts every five > minutes until it se

[ceph-users] Re: multiple-domain for S3 on rgws with same ceph backend on one zone

>>> Hello, >>> We have functional ceph swarm with a pair of S3 rgw in front that uses >>> A.B.C.D domain to be accessed. >>> >>> Now a new client asks to have access using the domain : E.C.D, but to >>> already existing buckets. This is not a scenario discussed in the docs. >>> Apparently, looking

[ceph-users] Re: multiple-domain for S3 on rgws with same ceph backend on one zone

Den mån 22 feb. 2021 kl 15:27 skrev Simon Pierre DESROSIERS < simonpierre.desrosi...@montreal.ca>: > Hello, > > We have functional ceph swarm with a pair of S3 rgw in front that uses > A.B.C.D domain to be accessed. > > Now a new client asks to have access using the domain : E.C.D, but to > alread

[ceph-users] Re: using secondary OSDs for reading

Den ons 10 feb. 2021 kl 19:09 skrev Seena Fallah : > But I think they can have no recovery ops. > No, but they would still have client ops even if there was no backfills or recovery anywhere on the OSD. -- May the most significant bit of your life be positive. __

[ceph-users] Re: using secondary OSDs for reading

Den ons 10 feb. 2021 kl 18:05 skrev Seena Fallah : > I have the same question about when recovery is going to happen! I think > recovering from second and third OSD can lead to not impact client IO too > when the primary OSD has another recovery ops! > Those OSDs (the 2nd and 3rd) are obviously s

[ceph-users] Re: NVMe and 2x Replica

Den fre 5 feb. 2021 kl 07:38 skrev Pascal Ehlert : > Sorry to jump in here, but would you care to explain why the total disk > usage should stay under 60%? > This is not something I have heard before and a quick Google search > didn't return anything useful. > If you have 3 hosts with 3 drives ea

[ceph-users] Re: rados bench error after running vstart script- HELP PLEASE

Den tis 26 jan. 2021 kl 14:20 skrev Bobby : > well, you are right. I forgot to create the pool. I thought 'rbd' pool is > created by default. Now it works after creating it :-) It was on older releases, I think many old clusters have an unused "rbd" pool. -- May the most significant bit of your

[ceph-users] Re: How to copy an OSD from one failing disk to another one

"ceph osd set norebalance" "ceph osd set nobackfill" Add new OSD, set osd weight to 0 on old OSD unset the norebalance and nobackfill options, and the cluster will do it all for you. Den tis 8 dec. 2020 kl 13:13 skrev huxia...@horebdata.cn < huxia...@horebdata.cn>: > Hi, dear cephers, > > On

[ceph-users] Re: Increase number of objects in flight during recovery

Den tors 3 dec. 2020 kl 10:11 skrev Frank Schilder : > I have the opposite problem as discussed in "slow down keys/s in > recovery". I need to increase the number of objects in flight during > rebalance. It is already all remapped PGs in state backfilling, but it > looks like no more than 8 object

[ceph-users] Re: rbd image backup best practice

Den fre 27 nov. 2020 kl 23:21 skrev Marc Roos : > Is there a best practice or guide for backuping rbd images? > One would think that most things that apply to an iscsi mounted device would be equally valid for an RBD mount, so you might look into that for hints and tips on how to backup remote ne

[ceph-users] Re: one osd down / rgw damoen wont start

Den fre 20 nov. 2020 kl 10:17 skrev Bernhard Krieger : > Hello, > > today i came across a strange behaviour. > After stoppping an osd, im not able to restart or /stop/start a radosgw > daemon. > The boot proccess will stuck until i have started the osd again. > > > Specs: > 3 ceph nodes > What is

[ceph-users] Re: Ceph EC PG calculation

Den ons 18 nov. 2020 kl 04:59 skrev Szabo, Istvan (Agoda) < istvan.sz...@agoda.com>: > I have this error: > I have 36 osd and get this: > Error ERANGE: pg_num 4096 size 6 would mean 25011 total pgs, which > exceeds max 10500 (mon_max_pg_per_osd 250 * num_in_osds 42) > I have 4:2 data EC pool, and

[ceph-users] Re: (Ceph Octopus) Repairing a neglected Ceph cluster - Degraded Data Reduncancy, all PGs degraded, undersized, not scrubbed in time

> > However I'm interested in the following > > On 11/16/20 11:31 AM, Janne Johansson wrote: > > So while one could always say "one more drive is better than your > > amount", there are people losing data with repl=2 or K+1 because some > > mor

[ceph-users] Re: (Ceph Octopus) Repairing a neglected Ceph cluster - Degraded Data Reduncancy, all PGs degraded, undersized, not scrubbed in time

Den mån 16 nov. 2020 kl 10:54 skrev Hans van den Bogert < hansbog...@gmail.com>: > > With this profile you can only loose one OSD at a time, which is really > > not that redundant. > That's rather situation dependent. I don't have really large disks, so > the repair time isn't that large. > Furthe

[ceph-users] Re: Beginner's installation questions about network

Den fre 13 nov. 2020 kl 21:50 skrev E Taka <0eta...@gmail.com>: > Hi Stefan, the cluster network has its own switch and is faster than the > public network. > Thanks for pointing me to the documentation. I must have overlooked this > sentence. > > But let me ask another question: do the OSD use th

[ceph-users] Re: question about rgw delete speed

Den ons 11 nov. 2020 kl 21:42 skrev Adrian Nicolae < adrian.nico...@rcs-rds.ro>: > Hey guys, > - 6 OSD servers with 36 SATA 16TB drives each and 3 big NVME per server > (1 big NVME for every 12 drives so I can reserve 300GB NVME storage for > every SATA drive), 3 MON, 2 RGW with Epyc 7402p and 128

[ceph-users] Re: Cephfs Kernel client not working properly without ceph cluster IP

Den tis 10 nov. 2020 kl 11:13 skrev Amudhan P : > Hi Nathan, > > Kernel client should be using only the public IP of the cluster to > communicate with OSD's. > "ip of the cluster" is a bit weird way to state it. A mounting client needs only to talk to ips in the public range yes, but OSDs always

[ceph-users] Re: frequent Monitor down

Den tors 29 okt. 2020 kl 20:16 skrev Tony Liu : > Typically, the number of nodes is 2n+1 to cover n failures. > It's OK to have 4 nodes, from failure covering POV, it's the same > as 3 nodes. 4 nodes will cover 1 failure. If 2 nodes down, the > cluster is down. It works, just not make much sense.

[ceph-users] Re: Ceph not showing full capacity

Den sön 25 okt. 2020 kl 15:18 skrev Amudhan P : > Hi, > > For my quick understanding How PG's are responsible for allowing space > allocation to a pool? > An objects name will decide which PG (from the list of PGs in the pool) it will end up on, so if you have very few PGs, the hashed/pseudorando

[ceph-users] Re: radosgw bucket subdomain with tls

Den tors 15 okt. 2020 kl 14:57 skrev Hans van den Bogert < hansbog...@gmail.com>: > To my knowledge that's a S3 client setting only. Mostly referred to as > 'path-style'. Some clients will call this "v2 style logins" or something to that effect. -- May the most significant bit of your life be

[ceph-users] Re: Error "Operation not permitted" using rbd pool init command

Den tors 8 okt. 2020 kl 10:25 skrev floda : > Hi guys, > I run the commands as the Linux root user and as the Ceph user > client.admin (I have turned off apparmor and other hardening things as > well). The chep user client.admin has the following setup in its keyring: > [client.admin] > key =

[ceph-users] Re: Slow ops on OSDs

Den tis 6 okt. 2020 kl 11:13 skrev Kristof Coucke : > I'm now wondering what my options are to improve the performance... The > main goal is to use the system again, and make sure write operations are > not affected. > - Putting weight on 0 for the slow OSDs (temporary)? This way they recovery > c

[ceph-users] Re: Consul as load balancer

Den tis 6 okt. 2020 kl 08:37 skrev Szabo, Istvan (Agoda) < istvan.sz...@agoda.com>: > > Hi, > Is there anybody tried consul as a load balancer? > Any experience? For rgw, load balancing is quite simple, and I guess almost any LB would work. The only major thing we have hit is that for AWS4 auth, y

[ceph-users] Re: rgw snapshots/backup

Den fre 2 okt. 2020 kl 14:07 skrev Manuel Lausch : > Hi, > we evaluate rados gateway to provide S3 Storage. > There is one question related to backup/snapshots. > Is there any way to create snapshots of buckets and or backup a bucket? > And how we can access data of a sapshot? > > I found only som

[ceph-users] Re: Understanding what ceph-volume does, with bootstrap-osd/ceph.keyring, tmpfs

Den mån 21 sep. 2020 kl 16:15 skrev Marc Roos : > When I create a new encrypted osd with ceph volume[1] > > Q4: Where is this luks passphrase stored? > I think the OSD asks the mon for it after auth:ing, so "in the mon DBs" somewhere. -- May the most significant bit of your life be positive. __

[ceph-users] Re: Migration to ceph.readthedocs.io underway

Den tors 17 sep. 2020 kl 12:09 skrev Lenz Grimmer : > > https://bootstrap-datepicker.readthedocs.io/en/v1.9.0/ > > Support Read the Docs! > > > > Please help keep us sustainable by allowing our Ethical Ads in your ad > > blocker or go ad-free by subscribing. > > Thanks for the info! That prompted

[ceph-users] Re: Benchmark WAL/DB on SSD and HDD for RGW RBD CephFS

Den ons 16 sep. 2020 kl 06:27 skrev Danni Setiawan < danni.n.setia...@gmail.com>: > Hi all, > > I'm trying to find performance penalty with OSD HDD when using WAL/DB in > faster device (SSD/NVMe) vs WAL/DB in same device (HDD) for different > workload (RBD, RGW with index bucket in SSD pool, and C

[ceph-users] Re: RGW bucket sync

Den ons 9 sep. 2020 kl 10:06 skrev Eugen Block : > Hi *, > > I'm wondering about what actually happens in the ceph cluster if I > copy/sync the content of one bucket into a different bucket. > > How does this work? It seems as if there's (almost) no client traffic > (except for the cp command, of

[ceph-users] Re: ceph pgs inconsistent, always the same checksum

I googled "got 0x6706be76, expected" and found some hits regarding ceph, so whatever it is, you are not the first, and that number has some internal meaning. Redhat solution for similar issue says that checksum is for seeing all zeroes, and hints at a bad write cache on the controller or something

[ceph-users] Re: OSDs get full with bluestore logs

Den fre 28 aug. 2020 kl 11:47 skrev Khodayar Doustar : > I've actually destroyed the cluster and a new one installed. > I've just changed the installation method and version. I've used > ceph-ansible this time and installed Nautilus. > The cluster worked fine with the same hardware. > Yes Janne, y

[ceph-users] Re: Fwd: Ceph Upgrade Issue - Luminous to Nautilus (14.2.11 ) using ceph-ansible

> > > partitions after checking disk partitions and whoami information. After > manually mounting the osd.108, now it's throwing permission error which I'm > still reviewing (bdev(0xd1be000 /var/lib/ceph/osd/ceph-108/block) open open > got: (13) Permission denied). Enclosed the log of the OSD fo

[ceph-users] Re: cephfs needs access from two networks

Den ons 26 aug. 2020 kl 14:16 skrev Simon Sutter : > Hello, > So I know, the mon services can only bind to just one ip. > But I have to make it accessible to two networks because internal and > external servers have to mount the cephfs. > The internal ip is 10.99.10.1 and the external is some publ

[ceph-users] Re: does ceph RBD have the ability to load balance?

Den fre 21 aug. 2020 kl 23:46 skrev 差很多先生 <948355...@qq.com>: > Hello Cephers, > I am a newcomer to Ceph.I know that RBD is a distributed block > storage device and libvirt supports rbd pool type. Suppose I have > a ceph cluster and use it to build a rbd pool to store virtual machine > images. Whe

[ceph-users] Re: does ceph rgw has any option to limit bandwidth

Apart from Marc Roos' reply, it seems like something that could be easily done with any normal kind of network rate limiting, so anything that limits traffic that matches "outgoing tcp from self with source-port 80,443*" should work for your rgw too, if you think it eats too much BW, without limiti

[ceph-users] Re: OSDs get full with bluestore logs

It says: FAILED assert(0 == "bluefs enospc") Could it be that the OSD disks you use are very very small? Den mån 17 aug. 2020 kl 20:26 skrev Khodayar Doustar : > Hi, > > I have a 3 node cluster of mimic with 9 osds (3 osds on each node). > I use this cluster to test integration of an applicati

[ceph-users] Re: radosgw health check url

Den sön 16 aug. 2020 kl 21:18 skrev Marc Roos : > I have just put radosgw behind haproxy, and wondered if there is a > health check endpoint. If not maybe it should be Yes, very good suggestion, both for having something where the rgw could actually state if it "feels mostly ok" in some decent f

[ceph-users] Re: Usable space vs. Overhead

Den ons 29 juli 2020 kl 16:34 skrev David Orman : > Thank you, everyone, for the help. I absolutely was mixing up the two, > which is why I was asking for guidance. The example made it clear. The > question I was trying to answer was: what would the capacity of the cluster > be, for actual data, b

[ceph-users] Re: Usable space vs. Overhead

Den ons 29 juli 2020 kl 03:17 skrev David Orman : > That's what the formula on the ceph link arrives at, a 2/3 or 66.66% > overhead. But if a 4 byte object is split into 4x1 byte chunks data (4 > bytes total) + 2x 1 byte chunks parity (2 bytes total), you arrive at 6 > bytes, which is 50% more tha

[ceph-users] Re: Current best practice for migrating from one EC profile to another?

Den tis 28 juli 2020 kl 18:50 skrev David Orman : > Hi, > > As we expand our cluster (adding nodes), we'd like to take advantage of > better EC profiles enabled by higher server/rack counts. I understand, as > Ceph currently exists (15.2.4), there is no way to live-migrate from one EC > profile to

[ceph-users] Re: ceph/rados performace sync vs async

Den lör 18 juli 2020 kl 02:18 skrev : > Daniel; > As I said, I don't actually KNOW most of this. > Seems correct in my view though. > As such, what I laid out was conceptual. > Ceph would need to be implemented to perform these operations in parallel, > or not. Conceptually, those areas where

[ceph-users] Re: Nautilus upgrade HEALTH_WARN legacy tunables

Den sön 5 juli 2020 kl 00:15 skrev Anthony D'Atri : > min_compat is a different thing entirely. > You need to set the tunables as a group. This will cause data to move, so > you may wish to throttle recovery, model the PG movement ahead of time, use > the upmap trick to control movement etc. > >

[ceph-users] Re: YUM doesn't find older release version of nautilus

Den tors 2 juli 2020 kl 14:42 skrev Lee, H. (Hurng-Chun) < h@donders.ru.nl>: > Hi, > > We use the official Ceph RPM repository (http://download.ceph.com/rpm- > nautilus/el7 ) for installing > packages on the client nodes running > CentOS7. > > But we

[ceph-users] Re: [RGW] Space usage vastly overestimated since Octopus upgrade

Den mån 29 juni 2020 kl 17:27 skrev Liam Monahan : > > For example, here is a bucket that all of a sudden reports that it has > 18446744073709551615 objects! The actual count should be around 20,000. > > "rgw.none": { > "size": 0, > "size_actual": 0, >

[ceph-users] Re: fault tolerant about erasure code pool

Den fre 26 juni 2020 kl 10:32 skrev Zhenshi Zhou : > Hi all, > > I'm going to deploy a cluster with erasure code pool for cold storage. > There are 3 servers for me to set up the cluster, 12 OSDs on each server. > Does that mean the data is secure while 1/3 OSDs of the cluster is down, > or only 2

[ceph-users] Re: Calculate recovery time

Den ons 17 juni 2020 kl 22:48 skrev Seena Fallah : > Yes I know but any point of view for backfill or priority used in Ceph > when recovering? > Client traffic defaults to highest prio, then scrubs+recovery, then rebalancing of misplaced PGs, if I recall correctly. Exception would be if you have

<    1   2   3   4   >