Re: [ceph-users] Consumer-grade SSD in Ceph

>> SATA: Micron 5100-5200-5300, Seagate Nytro 1351/1551 (don't forget to >> disable their cache with hdparm -W 0) We didn’t find a measurable difference doing this on 5100s, ymmv. Depending on your use-case, CRUSH rules (EC vs R), etc. sub-DWPD models may be fine for OSDs, but I suggest higher

Re: [ceph-users] ceph-users Digest, Vol 83, Issue 18

> Hi Sinan, > > I would not recommend using 860 EVO or Crucial MX500 SSD's in a Ceph cluster, > as those are consumer grade solutions and not enterprise ones. The OP knows that, but wants to know why. > Performance and durability will be issues. If feasible, I would simply go > NVMe as it

Re: [ceph-users] PG Balancer Upmap mode not working

> How is that possible? I dont know how much more proof I need to present that > there's a bug. FWIW, your pastes are hard to read with all the ? in them. Pasting non-7-bit-ASCII? > |I increased PGs and see no difference. From what pgp_num to what new value? Numbers that are not a power of

Re: [ceph-users] WAL/DB size

WAL, see >> http://yourcmc.ru/wiki/Ceph_performance#About_block.db_sizing >> >> 15 августа 2019 г. 1:15:58 GMT+03:00, Anthony D'Atri >> пишет: >>> >>> Good points in both posts, but I think there’s still some unclarity. _

Re: [ceph-users] WAL/DB size

Good points in both posts, but I think there’s still some unclarity. Absolutely let’s talk about DB and WAL together. By “bluestore goes on flash” I assume you mean WAL+DB? “Simply allocate DB and WAL will appear there automatically” Forgive me please if this is obvious, but I’d like to see a

Re: [ceph-users] How to maximize the OSD effective queue depth in Ceph?

> However, I'm starting to think that the problem isn't with the number > of threads that have work to do... the problem may just be that the > OSD & PG code has enough thread locking happening that there is no > possible way to have more than a few things happening on a single OSD > (or perhaps a

Re: [ceph-users] How to add 100 new OSDs...

>>> We have been using: >>> >>> osd op queue = wpq >>> osd op queue cut off = high >>> >>> It virtually eliminates the impact of backfills on our clusters. Our > > It does better because it is a fair share queue and doesn't let recovery > ops take priority over client ops at any point for any

Re: [ceph-users] How to add 100 new OSDs...

Paul Emmerich wrote: > +1 on adding them all at the same time. > > All these methods that gradually increase the weight aren't really > necessary in newer releases of Ceph. Because the default backfill/recovery values are lower than they were in, say, Dumpling? Doubling (or more) the size of

Re: [ceph-users] New best practices for osds???

> This is worse than I feared, but very much in the realm of concerns I > had with using single-disk RAID0 setups.? Thank you very much for > posting your experience!? My money would still be on using *high write > endurance* NVMes for DB/WAL and whatever I could afford for block.? yw. Of

Re: [ceph-users] New best practices for osds???

> We run few hundred HDD OSDs for our backup cluster, we set one RAID 0 per HDD > in order to be able > to use -battery protected- write cache from the RAID controller. It really > improves performance, for both > bluestore and filestore OSDs. Having run something like 6000 HDD-based FileStore

Re: [ceph-users] Observation of bluestore db/wal performance

This may be somewhat controversial, so I’ll try to tread lightly. Might we infer that your OSDs are on spinners? And at 500 GB it would seem likely that they and the servers are old? Please share hardware details and OS. Having suffered an “enterprise” dogfood deployment in which I had to

Re: [ceph-users] clock skew

> @Janne: i will checkout/implement the peer config per your suggestion. > However what confuses us is that chrony thinks the clocks match, and > only ceph feels it doesn't. So we are not sure if the peer config will > actually help in this situation. But time will tell. Ar ar. Chrony thinks

Re: [ceph-users] rgw, nss: dropping the legacy PKI token support in RadosGW (removed in OpenStack Ocata)

I've been away from OpenStack for a couple of years now, so this may have changed. But back around the Icehouse release, at least, upgrading between OpenStack releases was a major undertaking, so backing an older OpenStack with newer Ceph seems like it might be more common than one might

Re: [ceph-users] Ceph block storage cluster limitations

> Hello, > > I wanted to know if there are any max limitations on > > - Max number of Ceph data nodes > - Max number of OSDs per data node > - Global max on number of OSDs > - Any limitations on the size of each drive managed by OSD? > - Any limitation on number of client nodes? > - Any

Re: [ceph-users] [Bluestore] Some of my osd's uses BlueFS slow storage for db - why?

> Date: Fri, 22 Feb 2019 16:26:34 -0800 > From: solarflow99 > > > Aren't you undersized at only 30GB? I thought you should have 4% of your > OSDs The 4% guidance is new. Until relatively recently the oft-suggested and default value was 1%. > From: "Vitaliy Filippov" > Numbers are easy to

Re: [ceph-users] Ceph cluster stability

? Did we start recommending that production mons run on a VM? I'd be very hesitant to do that, though probably some folks do. I can say for sure that in the past (Firefly) I experienced outages related to mons running on HDDs. That was a cluster of 450 HDD OSDs with colo journals and

Re: [ceph-users] ceph mon_data_size_warn limits for large cluster

On older releases, at least, inflated DBs correlated with miserable recovery performance and lots of slow requests. The DB and OSDs were also on HDD FWIW. A single drive failure would result in substantial RBD impact. > On Feb 18, 2019, at 3:28 AM, Dan van der Ster wrote: > > Not

Re: [ceph-users] Questions about using existing HW for PoC cluster

> Been reading "Learning Ceph - Second Edition” An outstanding book, I must say ;) > So can I get by with using a single SATA SSD (size?) per server for RocksDB / > WAL if I'm using Bluestore? Depends on the rest of your setup and use-case, but I think this would be a bottleneck. Some

Re: [ceph-users] Ceph Nautilus Release T-shirt Design

>> Lenz has provided this image that is currently being used for the 404 >> page of the dashboard: >> >> https://github.com/ceph/ceph/blob/master/src/pybind/mgr/dashboard/frontend/src/assets/1280px-Nautilus_Octopus.jpg > > Nautilus *shells* are somewhat iconic/well known/distinctive. Maybe a >

Re: [ceph-users] Questions re mon_osd_cache_size increase

Thanks, Greg. This is as I suspected. Ceph is full of subtleties and I wanted to be sure. -- aad > > The osd_map_cache_size controls the OSD’s cache of maps; the change in 13.2.3 > is to the default for the monitors’. > On Mon, Jan 7, 2019 at 8:24 AM Anthony D'Atr

[ceph-users] Questions re mon_osd_cache_size increase

> * The default memory utilization for the mons has been increased > somewhat. Rocksdb now uses 512 MB of RAM by default, which should > be sufficient for small to medium-sized clusters; large clusters > should tune this up. Also, the `mon_osd_cache_size` has been > increase from 10

Re: [ceph-users] Bluestore nvme DB/WAL size

> It'll cause problems if yours the only one NVMe drive will die - you'll lost > all the DB partitions and all the OSDs are going to be failed The severity of this depends a lot on the size of the cluster. If there are only, say, 4 nodes total, for sure the loss of a quarter of the OSDs will

[ceph-users] warning: fast-diff map is invalid operation may be slow; object map invalid

We turned on all the RBD v2 features while running Jewel; since then all clusters have been updated to Luminous 12.2.2 and additional clusters added that have never run Jewel. Today I find that a few percent of volumes in each cluster have issues, examples below. I'm concerned that these

Re: [ceph-users] Favorite SSD

> Micron 5200 line seems to not have a high endurance SKU like the 5100 line > sadly. The 3.84TB 5200 PRO is rated at ~2.4 DWPD — you need higher than that? I do find references to higher-durability ~5DWPD 5200 MAX models up to 1.9 TB. Online resources on the 5200 product line don’t always

Re: [ceph-users] Adding node efficient data move.

> When adding a node and I increment the crush weight like this. I have > the most efficient data transfer to the 4th node? > > sudo -u ceph ceph osd crush reweight osd.23 1 > sudo -u ceph ceph osd crush reweight osd.24 1 > sudo -u ceph ceph osd crush reweight osd.25 1 > sudo -u ceph ceph osd

Re: [ceph-users] Exact scope of OSD heartbeating?

Thanks, Dan. I thought so but wanted to verify. I'll see if I can work up a doc PR to clarify this. >> The documentation here: >> >> http://docs.ceph.com/docs/master/rados/configuration/mon-osd-interaction/ >> >> says >> >> "Each Ceph OSD Daemon checks the heartbeat of other Ceph OSD

[ceph-users] Exact scope of OSD heartbeating?

The documentation here: http://docs.ceph.com/docs/master/rados/configuration/mon-osd-interaction/ says "Each Ceph OSD Daemon checks the heartbeat of other Ceph OSD Daemons every 6 seconds" and " If a neighboring

Re: [ceph-users] ceph-deploy: recommended?

> ?I read a couple of versions ago that ceph-deploy was not recommended > for production clusters.? InkTank had sort of discouraged the use of ceph-deploy; in 2014 we used it only to deploy OSDs. Some time later the message changed. ___ ceph-users

Re: [ceph-users] IO rate-limiting with Ceph RBD (and libvirt)

> FYI: I/O limiting in combination with OpenStack 10/12 + Ceph doesn?t work > properly. Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1476830 > That's an OpenStack bug, nothing to do with Ceph. Nothing stops you from using virsh to

Re: [ceph-users] Migrating to new pools

Cumulative followup to various insightful replies. I wrote: No, it's not really possible currently and we have no plans to add >>> such support since it would not be of any long-term value. >> >> The long-term value would be the ability to migrate volumes from, say, a >> replicated pool

Re: [ceph-users] Migrating to new pools

>> I was thinking we might be able to configure/hack rbd mirroring to mirror to >> a pool on the same cluster but I gather from the OP and your post that this >> is not really possible? > > No, it's not really possible currently and we have no plans to add > such support since it would not be of

Re: [ceph-users] Sizing your MON storage with a large cluster

Thanks, Wido -- words to live by. I had all kinds of problems with mon DBs not compacting under Firefly, really pointed out the benefit of having ample space on the mons -- and the necessity of having those DB's live on something faster than an LFF HDD. I've had this happen when using

Re: [ceph-users] HW Raid vs. Multiple OSD

Oscar, a few thoughts: o I think you might have some misunderstandings about how Ceph works. Ceph is best deployed as a single cluster spanning multiple servers, generally at least 3. Is that your plan? It sort of sounds as though you're thinking of Ceph managing only the drives local to

[ceph-users] Anyone else having digest issues with Apple Mail?

For a couple of weeks now digests have been appearing to me off and on with a few sets of MIME headers and maybe 1-2 messages. When I look at the raw text the whole digest is in there. Screencap below. Anyone else experiencing this?

[ceph-users] Vote re release cadence

One vote for: * Drop the odd releases, and aim for a ~9 month cadence. This splits the difference between the current even/odd pattern we've been doing. We've already been bit by gotchas with upgrades even between point releases, so I favor strategies that limit the number of upgrade paths in

Re: [ceph-users] High iowait on OSD node

My first suspicion would be the HBA. Are you using a RAID HBA? If so I suggest checking the status of your BBU/FBWC and cache policy. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Crushmap from Rack aware to Node aware

All very true and worth considering, but I feel compelled to mention the strategy of setting mon_osd_down_out_subtree_limit carefully to prevent automatic rebalancing. *If* the loss of a failure domain is temporary, ie. something you can fix fairly quickly, it can be preferable to not start

Re: [ceph-users] Re-weight Entire Cluster?

up 1.0 1.0 > 221 1.0 osd.221 up 1.0 1.0 > 222 1.0 osd.222 up 1.0 1.0 > 223 1.0 osd.223 up 1.0 1.0 > > -Original Message-

Re: [ceph-users] Re-weight Entire Cluster?

> It appears the current best practice is to weight each OSD according to it?s > size (3.64 for 4TB drive, 7.45 for 8TB drive, etc). OSD’s are created with those sorts of CRUSH weights by default, yes. Which is convenient, but it’s import to know that those weights are arbitrary, and what

Re: [ceph-users] Sharing SSD journals and SSD drive choice

At a meeting with Intel folks a while back, they discussed the idea that future large devices — which we’re starting to now see — would achieve greater *effective* durability via a lower cost/GB that encourages the use of larger than needed devices. Which is a sort of overprovisioning, just

Re: [ceph-users] SSD Primary Affinity

Re ratio, I think you’re right. Write performance depends for sure on what the journal devices are. If the journals are colo’d on spinners, then for sure the affinity game isn’t going to help writes massively. My understanding of write latency is that min_size journals have to be written

Re: [ceph-users] SSD Primary Affinity

I get digests, so please forgive me if this has been covered already. > Assuming production level, we would keep a pretty close 1:2 SSD:HDD ratio, 1:4-5 is common but depends on your needs and the devices in question, ie. assuming LFF drives and that you aren’t using crummy journals. > First

Re: [ceph-users] What's the actual justification for min_size?

several major LTS releases now. :) > So: just don't ever set min_size to 1. > -Greg > On Tue, Mar 21, 2017 at 6:04 PM Anthony D'Atri <a...@dreamsnake.net> wrote: > >> a min_size of 1 is dangerous though because it means you are 1 hard disk > >> failure away from losi

Re: [ceph-users] add multiple OSDs to cluster

Deploying or removing OSD’s in parallel for sure can save elapsed time and avoid moving data more than once. There are certain pitfalls, though, and the strategy needs careful planning. - Deploying a new OSD at full weight means a lot of write operations. Running multiple whole-OSD backfills

Re: [ceph-users] What's the actual justification for min_size?

>> a min_size of 1 is dangerous though because it means you are 1 hard disk >> failure away from losing the objects within that placement group entirely. a >> min_size of 2 is generally considered the minimum you want but many people >> ignore that advice, some wish they hadn't. > > I admit I

Re: [ceph-users] Directly addressing files on individual OSD

As I parse Youssef’s message, I believe there are some misconceptions. It might help if you could give a bit more info on what your existing ‘cluster’ is running. NFS? CIFS/SMB? Something else? 1) Ceph regularly runs scrubs to ensure that all copies of data are consistent. The checksumming

[ceph-users] apologies for the erroneous subject - should have been Re: Unable to boot OS on cluster node

A certain someone bumped my elbow as I typed, think in terms of this week’s family-bombed video going the rounds on FB. My ignominy is boundless and my door now locked when replying. — aad ___ ceph-users mailing list ceph-users@lists.ceph.com

[ceph-users] http://www.dell.com/support/home/us/en/04/product-support/servicetag/JFGQY02/warranty#

> As long as you don?t nuke the OSDs or the journals, you should be OK. This. Most HBA failures I’ve experienced don’t corrupt data on the drives, bit it can happen. Assuming the data is okay, you should be able to just install the OS, install the *same version* of Ceph packages, reboot, and

Re: [ceph-users] mon.mon01 store is getting too big! 18119 MB >= 15360 MB -- 94% avail

> In particular, when using leveldb, stalls while reading or writing to > the store - typically, leveldb is compacting when this happens. This > leads to all sorts of timeouts to be triggered, but the really annoying > one would be the lease timeout, which tends to result in flapping quorum. >

Re: [ceph-users] cephfs ata1.00: status: { DRDY }

YMMV of course, but the first thing that struck me was the constraint of scrub times. Constraining them to fewer hours can mean that more run in parallel. If you truly have off-hours for client ops (Graphite / Grafana are great for visualizing that) that might make sense, but in my 24x7

Re: [ceph-users] Intel P3700 SSD for journals

to PERC 730 raid controller? > > Is there anything particular required when installing the CentOS on > these drives, or they will be automatically detected and work out of > the box by default? Thx will > > On Mon, Nov 21, 2016 at 12:16 PM, Anthony D'Atri <a...@dreamsnake.net> wr

Re: [ceph-users] Intel P3700 SSD for journals

The SATA S3700 series has been the de-facto for journals for some time. And journals don’t neeed all that much space. We’re using 400GB P3700’s. I’ll say a couple of things: o Update to the latest firmware available when you get your drives, qual it and stick with it for a while so you have

Re: [ceph-users] Monitoring Overhead

> Collectd and graphite look really nice. Also look into Grafana, and of course RHSC. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Upgrade from Hammer to Jewel

> We have an Openstack which use Ceph for Cinder and Glance. Ceph is in > Hammer release and we need to upgrade to Jewel. My question is : > are the Hammer clients compatible with the Jewel servers ? (upgrade of Mon > then Ceph servers first) > As the upgrade of the Ceph client need a reboot of

Re: [ceph-users] Terrible RBD performance with Jewel

> FWIW, the xfs -n size=64k option is probably not a good idea. Agreed, moreover it’s a really bad idea. You get memory allocation slowdowns as described in the linked post, and eventually the OSD dies. It can be mitigated somewhat by periodically (say every 2 hours, ymmv) flushing the

Re: [ceph-users] Help recovering failed cluster

> Current cluster health: >cluster 537a3e12-95d8-48c3-9e82-91abbfdf62e0 > health HEALTH_WARN >5 pgs degraded >8 pgs down >48 pgs incomplete >3 pgs recovering >1 pgs recovery_wait >76 pgs stale >5 pgs stuck

Re: [ceph-users] Journal partition owner's not change to ceph

> The GUID for a CEPH journal partition should be > "45B0969E-9B03-4F30-B4C6-B4B80CEFF106" > I haven't been able to find this info in the documentation on the ceph site The GUID typecodes are listed in the /usr/sbin/ceph-disk script. I had an issue a couple years ago where a subset of OSD’s in

Re: [ceph-users] Jewel ubuntu release is half cooked

Re: > 2. Inefficient chown documentation - The documentation states that one should > "chown -R ceph:ceph /var/lib/ceph" if one is looking to have ceph-osd ran as > user ceph and not as root. Now, this command would run a chown process one > osd at a time. I am considering my cluster to be a

Re: [ceph-users] Do you see a data loss if a SSD hosting several OSD journals crashes

nly min_size before > ack? > > But for me the takeaway is that writes are protected - even if the journal > drive crashes, I am covered. > > - epk > > -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Anthony D'

Re: [ceph-users] Do you see a data loss if a SSD hosting several OSD journals crashes

> Ceph will not acknowledge a client write before all journals (replica > size, 3 by default) have received the data, so loosing one journal SSD > will NEVER result in an actual data loss. Some say that all replicas must be written; others say that only min_size, 2 by default, must be written

Re: [ceph-users] dense storage nodes

[ too much to quote ] Dense nodes often work better for object-focused workloads than block-focused, the impact of delayed operations is simply speed vs. a tenant VM crashing. Re RAID5 volumes to decrease the number of OSD’s: This sort of approach is getting increasing attention in that it

Re: [ceph-users] SSD Journal

> Right now we run the journal as a partition on the data disk. I've build > drives without journals and the write performance seems okay but random io > performance is poor in comparison to what it should be. Co-located journals have multiple issues: o The disks are presented with double

Re: [ceph-users] Scrub Error / How does ceph pg repair work?

For me that's true about 1/3 the time, but often I do still have to repair the PG after removing the affected OSD. YMMV. Agree that 99+% of the inconsistent PG's I see correlate directly to disk flern. Check /var/log/kern.log*, /var/log/messages*, etc. and I'll bet you find errors

Re: [ceph-users] Scrub Error / How does ceph pg repair work?

Agree that 99+% of the inconsistent PG's I see correlate directly to disk flern. Check /var/log/kern.log*, /var/log/messages*, etc. and I'll bet you find errors correlating. -- Anthony ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] Sunday's Ceph based business model

Interesting idea. I'm not sure, though, Ceph is designed with this sort of latency in mind. Crashplan does let you do something very similar for free, as I understand it, though it's more of a nearline thing. ___ ceph-users mailing list

Re: [ceph-users] adding osd node best practice

We have application cluster and ceph as storage solution, cluster consists of six servers, so we've installed monitor on every one of them, to have ceph cluster sane (quorum) if server or two of them goes down. You want an odd number for sure, to avoid the classic split-brain problem:

Re: [ceph-users] adding osd node best practice

1) That's an awful lot of mons. Are they VM's or something? My sense is that mons 5 have diminishing returns at best. 2) Only two OSD nodes? Assume you aren't running 3 copies of data or racks. 3) The new nodes will have fewer OSD's? Be careful with host / OSD weighting to avoid a

Re: [ceph-users] No auto-mount of OSDs after server reboot

One thing than can cause this is messed-up partition ID's / typecodes. Check out the ceph-disk script to see how they get applied. I have a few systems that somehow got messed up -- at boot they don't get started, but if I mounted them manually on /mnt, checked out the whoami file and

[ceph-users] Survey re journals on SSD vs co-located on spinning rust

My apologies if this has been covered ad-naseum in the past; I wasn't finding a lot of relevant archived info. I'm curious how may people are using 1) OSD's on spinning disks, with journals on SSD's -- how many journals per SSD? 4-5? 2) OSD's on spinning disks, with [10GB] journals