[ceph-users] Re: Mgr stability

15, 2019, at 8:10 AM, Mykola Golub wrote: > > On Wed, Aug 14, 2019 at 12:12:36PM -0500, Reed Dier wrote: > >> My main metrics source is the influx plugin, but I enabled the >> prometheus plugin to get access to the per-rbd image metrics. I may >> disable prometheus

[ceph-users] Mgr stability

Hi all, I'm experiencing some stability issues with my ceph-mgr daemons post Nautilus upgrade (13.2.6->14.2.2). On top of Ubuntu 18.04, the mgr's are colocated with the mon's, and 2 of the 3 mon/mgr nodes also colocate an MDS, running a single rank active:standby-replay. The main issue(s) I am

[ceph-users] Re: Mgr stability

s://tracker.ceph.com/issues/41171 > > # ceph mgr module ls > { >"enabled_modules": [ >"balancer", >"crash", >"dashboard", > "restful", >"status", >"zabbi

[ceph-users] Crush device class switchover

I currently have two roots in my crush map, one for HDD devices and one for SSD devices, and have had it that way since Jewel. I am currently on Nautilus, and have had my crush device classes for my OSD's set since Luminous. > ID CLASS WEIGHTTYPE NAME > -13 105.37599 root ssd > -11

[ceph-users] Re: mgr daemons becoming unresponsive

Do you also have the balancer module on? I experienced extremely bad stability issues where the MGRs would silently die with the balancer module on. And by on, I mean 'active:true` by way of `ceph balancer on`. Once I disabled the automatic balancer, it seemed to become much more stable. I can

[ceph-users] Re: zabbix sender issue with v14.2.5

Piggybacking on this thread to say that I too am having the same behavior. Ubuntu 18.04 zabbix-sender 4.4.3-1+bionic ceph-mgr14.2.5-1bionic I am still getting all metrics in my Zabbix host, it is just the error being thrown by the ceph-mgr. Reed > On Dec 11,

[ceph-users] Re: zabbix sender issue with v14.2.5

Excellent sleuthing. I was able to change the key on my Zabbix server instance and all is happy in Ceph again. Thanks, Reed > On Dec 11, 2019, at 10:24 AM, Gary Molenkamp wrote: > > I dislike replying to my own post, but I found the issue: > > Looking at the changelog for 14.2.5, the

[ceph-users] Re: Recurring issue: PG is inconsistent, but lists no inconsistent objects

I had something slightly similar to you. However, my issue was specific/limited to the device_health_metrics pool that is auto-created with 1 PG when you turn that mgr feature on. https://www.mail-archive.com/ceph-users@lists.ceph.com/msg56315.html

[ceph-users] Re: OSD rebalancing issue - should drives be distributed equally over all nodes

Hi Thomas, How does your crush map/tree look? If your crush failure domain is by host, then your 96x 8T disks will be as useful as you're 1.6T disks, because smallest failure domain is your limiting factor. So you can either redistribute your disks to be 16x8T+32x1.6T per host, or you could

[ceph-users] Re: leftover: spilled over 128 KiB metadata after adding db device

Easiest way I know would be to use $ ceph tell osd.X compact This is what cures that whenever I have metadata spillover. Reed > On Mar 2, 2020, at 3:32 AM, Stefan Priebe - Profihost AG > wrote: > > Hello, > > i added a db device to my osds running nautilus. The DB data migratet > over some

[ceph-users] Re: moving small production cluster to different datacenter

I did this, but with the benefit of taking the network with me, just a forklift from one datacenter to the next. Shutdown the clients, then OSDs, then MDS/MON/MGRs, then switches. Reverse order back up, > On Jan 28, 2020, at 4:19 AM, Marc Roos wrote: > > > Say one is forced to move a

[ceph-users] Re: High CPU usage by ceph-mgr in 14.2.5

Chiming in to mirror this. 250 OSDs, and after 14.2.6 CPU is much lower on the mgr, and the balancer doesn't hang, which was the main thing that would stall previously. Reed > On Jan 16, 2020, at 4:30 AM, Dan van der Ster wrote: > > Hey Wido, > We upgraded a 550-osd cluster from 14.2.4 to

[ceph-users] Re: New to ceph / Very unbalanced cluster

Copying the ML, because I forgot to reply-all. Reed > On Apr 15, 2020, at 3:58 PM, Reed Dier wrote: > > The problem is almost certainly stemming from unbalanced OSD distribution > among your hosts, and assuming you are using a default 3x replication across > hosts crush rul

[ceph-users] Re: Recommendation for decent write latency performance from HDDs

Going to resurrect this thread to provide another option: LVM-cache, ie putting a cache device in-front of the bluestore-LVM LV. I only mention this because I noticed it in the SUSE documentation for SES6 (based on Nautilus) here: https://documentation.suse.com/ses/6/html/ses-all/lvmcache.html

[ceph-users] ceph-df free discrepancy

Hopefully someone can sanity check me here, but I'm getting the feeling that the MAX AVAIL in ceph df isn't reporting the correct value in 14.2.8 (mon/mgr/mds are .8, most OSDs are .7) > RAW STORAGE: > CLASS SIZEAVAIL USEDRAW USED %RAW USED > hdd 530

[ceph-users] Re: ceph-df free discrepancy

> On Apr 10, 2020, at 5:55 PM, Paul Emmerich wrote: > > On Sat, Apr 11, 2020 at 12:43 AM Reed Dier wrote: >> That said, as a straw man argument, ~380GiB free, times 60 OSDs, should be >> ~22.8TiB free, if all OSD's grew evenly, which they won't > > Yes, that's the prob

[ceph-users] Re: Can 16 server grade ssd's be slower then 60 hdds? (no extra journals)

=== > Read Policy = Yes > Write Policy = Yes > IO Policy = Yes > Access Policy = Yes > Disk Cache Policy = Yes (but only HDD’s in this case) > Reconstruction = Yes > Deny Locate = No > Deny CC = No > Allow Ctrl Encryption = No > Enable LDBBM = No > Support Fas

[ceph-users] Re: Can 16 server grade ssd's be slower then 60 hdds? (no extra journals)

rver. But > that’s a lot of work - not knowing that this will improve the speed a lot. > > I am using rbd with hyperconvergenced nodes (4 at the moment) pools are 2 and > 3 times replicated. actually the performance for windows and linux vms with > the hdd osd pool was ok. But w

[ceph-users] Re: Can 16 server grade ssd's be slower then 60 hdds? (no extra journals)

If using storcli/perccli for manipulating the LSI controller, you can disable the on-disk write cache with: storcli /cx/vx set pdcache=off You can also ensure that you turn off write caching at the controller level with storcli /cx/vx set iopolicy=direct storcli /cx/vx set wrcache=wt You can

[ceph-users] Re: Can 16 server grade ssd's be slower then 60 hdds? (no extra journals)

et more then 2000 > iops on single thread out of the ssds. I know thet ceph does not shine with > single thread. But 600 iops is not very much... > > philipp > > -Ursprüngliche Nachricht- > Von: Reed Dier > Gesendet: Dienstag, 01. September 2020 22:3

[ceph-users] Re: Upgrade options and *request for comment

Your options while staying on Xenial are only to Nautilus. In the below chart, X is provided by the Ceph repos, U denotes from Ubuntu repos. rel jewel luminus mimic nautilus octopus trusty X X xenial XU X X X bionic U X X X focal XU Octopus is only supported on bionic and focal. Xenial

[ceph-users] Re: The feasibility of mixed SSD and HDD replicated pool

Late reply, but I have been using what I refer to as a "hybrid" crush topology for some data for a while now. Initially with just rados objects, and later with RBD. We found that we were able to accelerate reads to roughly all-ssd performance levels, while bringing up the tail end of the write

[ceph-users] Re: Ceph Zabbix Monitoring : No such file or directory

Just to confirm, you did configure a trapper host in Zabbix, and then you pointed the mgr Zabbix sender to the Zabbix server (or proxy) with the same hostname? > $ ceph zabbix config-show | jq > { > "zabbix_port": 10051, > "zabbix_host": "$server_or_proxy_ip_or_fqdn", > "identifier":

[ceph-users] Re: High ceph_osd_commit_latency_ms on Toshiba MG07ACA14TE HDDs

Just throwing my hat in here with a small bit of anecdotal experience. In the early days of experimenting with ceph, I had 24x 8T disk, all behind RAID controllers as R0 vd's with no BBU (so controller cache is WT, default value), and pdcache (disk write cache) enabled (default value). We had

[ceph-users] Re: dealing with spillovers

I'm going to piggy back on this somewhat. I've battled RocksDB spillovers over the course of the life of the cluster since moving to bluestore, however I have always been able to compact it well enough. But now I am stumped at getting this to compact via $ceph tell osd.$osd compact, which has

[ceph-users] Re: dealing with spillovers

the ticket above? > > > > Thanks, > > Igor > > On 6/6/2020 1:07 AM, Reed Dier wrote: >> I'm going to piggy back on this somewhat. >> >> I've battled RocksDB spillovers over the course of the life of the cluster >> since moving to bluestore

[ceph-users] Re: dealing with spillovers

is unlikely to be correct... > > No more ideas but do data migration using ceph-bluestore-tool. > > I would appreciate if you share whether it helps in both short- and > long-term. Will this reappear or not? > > > Thanks, > > Igor > > > > On 6

[ceph-users] Re: Zabbix module Octopus 15.2.3

I'm going to resurrect this thread to throw my hat in the ring as I am having this issue as well. I just moved to 15.2.4 on Ubuntu 18.04/bionic, and Zabbix is 5.0.2. > $ ceph zabbix config-show > Error EINVAL: Traceback (most recent call last): > File "/usr/share/ceph/mgr/mgr_module.py", line

[ceph-users] Re: add server in crush map before osd

Just to piggyback on this, the below are the correct answers. However, how I do it, which is admittedly not the best way, but it is the easy way. I set the norecover, nobackfill flags I run my osd creation script against the first disk on the new host to make sure that everything is working

[ceph-users] Re: Debian repo for ceph-iscsi

I know this isn't what you asked for, but I do know that Canonical is building this package for focal and up. While not Buster, could possibly be a compromise to move things forward without huge plumbing changes between Debian and Ubuntu. You may also be able to hack and slash your way through

[ceph-users] Re: diskprediction_local fails with python3-sklearn 0.22.2

I'm going to resurrect this thread in hopes that in the 6 months since, someone has found a solution? After recently upgrading my mgr's to 20.04 and 15.2.8, the diskprediction_local module is failing for me in the exact same manner. > $ dpkg -l | grep sklearn > ii python3-sklearn

[ceph-users] Re: ceph PGs issues

.pl/view/eac56017 > > Our main struggle is when we try to rsync data, the rsync process hangs > because it encounters an inaccessible object. > > Is there a way we can take out the incomplete PGs to be able to copy data > smoothly without having to reset the rsync process

[ceph-users] Re: ceph PGs issues

You have incomplete PGs, which means you have inactive data, because the data isn't there. This will typically only happen when you have multiple concurrent disk failures, or something like that, so I think there is some missing info. >1 osds exist in the crush map but not in the

[ceph-users] Re: Manager carries wrong information until killing it

I don't have a solution to offer, but I've seen this for years with no solution. Any time a MGR bounces, be it for upgrades, or a new daemon coming online, etc, I'll see a scale spike like is reported below. Just out of curiosity, which MGR plugins are you using? I have historically used the

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

Consider this a +1 for pretty much every point that Matthew made, and articulated much better than I would have. I wholly understand the goal with making ceph more approachable for people learning it, and I think that is a great cause and deserves attention. I also think Tim Serong's

[ceph-users] XFS on RBD on EC painfully slow

Hoping someone may be able to help point out where my bottleneck(s) may be. I have an 80TB kRBD image on an EC8:2 pool, with an XFS filesystem on top of that. This was not an ideal scenario, rather it was a rescue mission to dump a large, aging raid array before it was too late, so I'm working

[ceph-users] Re: XFS on RBD on EC painfully slow

I guess I should probably have been more clear, this is one pool of many, so the other OSDs aren't idle. So I don't necessarily think that the PG bump would be the worst thing to try, but its definitely not as bad as I may have made it sound. Thanks, Reed > On May 27, 2021, at 11:37 PM,

[ceph-users] Re: XFS on RBD on EC painfully slow

, positive or negative. Thanks for the suggestion, Reed > On May 28, 2021, at 3:24 AM, Matthias Ferdinand wrote: > > On Thu, May 27, 2021 at 02:54:00PM -0500, Reed Dier wrote: >> Hoping someone may be able to help point out where my bottleneck(s) may be. >> >> I have an

[ceph-users] Re: [External Email] Re: XFS on RBD on EC painfully slow

Providing an overdue update to wrap this thread up. It turns out I wasn't seeing the forest for the trees. Parallelizing the copy did in fact yield much larger results than the single threaded copies. In the end we used a home-brewed python script to parallelize the copy using cp, rather than

[ceph-users] Re: Preferred order of operations when changing crush map and pool rules

I've not undertaken such a large data movement, The pgupmap script may be of use here, but assuming that its not. But if I were, I would first take many backups of the current crush map. I would set the noreblance and norecover flags. Then I would verify all of the backfill settings are as

[ceph-users] Re: Ceph server

I'm going to echo what Stefan said. I would ditch the 2x SATA drives to free up your slots. Replace with an M.2 or SATADOM. I would also recommend moving from the 2x X710-DA2 cards to 1x X710-DA4 card. It can't saturate the x8 slot, and it frees up a PCIe slot for possibly another NVMe card or

[ceph-users] Re: Best way to add OSDs - whole node or one by one?

I'm sure there is a "correct" way, but I think it mostly relates to how busy your cluster is, and how tolerant it is of the added load from the backfills. My current modus operandi is to set the noin, noout, nobackfill, norecover, and norebalance flags first. This makes sure that new OSDs don't

[ceph-users] Re: ceph-ansible in Pacific and beyond?

I too would be amenable to a cephadm orchestrator on bare metal if that were an option. I thought that I would drop this video for anyone that hasn't seen it yet. It hits on a lot of people's sentiments about containers being not great for debugging, while also going into some of the pros to

[ceph-users] Re: LVM vs. direct disk acess

I think this would be a great place in the ML to look. https://ceph-users.ceph.narkive.com/AthYx879/ceph-volume-migration-and-disk-partition-support Reed > On Mar 19, 2021, at 2:17 PM, Marc wrote:

[ceph-users] Re: Best way to add OSDs - whole node or one by one?

ut phase? > > Kind regards, > Caspar > > Op do 11 mrt. 2021 om 23:54 schreef Reed Dier : > >> I'm sure there is a "correct" way, but I think it mostly relates to how >> busy your cluster is, and how tolerant it is of the added load from the >> backfill

[ceph-users] Re: Has anyone contact Data for Samsung Datacenter SSD Support ?

Not a direct answer to your question, but it looks like Samsung's DC Toolkit may allow for user adjusted over-provisioning. https://www.samsung.com/semiconductor/global.semi.static/S190311-SAMSUNG-Memory-Over-Provisioning-White-paper.pdf

[ceph-users] Re: Ceph osd Reweight command in octopus

Have you tried a more aggressive reweight value? I've seen some stubborn crush maps that don't start moving date until 0.9 or lower in some cases. Reed > On Mar 11, 2021, at 10:29 AM, Brent Kennedy wrote: > > We have a ceph octopus cluster running 15.2.6, its indicating a near full > osd

[ceph-users] Re: HBA vs caching Raid controller

I don't have any performance bits to offer, but I do have one experiential bit to offer. My initial ceph deployment was on existing servers, that had LSI raid controllers (3108 specifically). We created R0 vd's for each disk, and had BBUs so were using write back caching. The big problem that

[ceph-users] Re: The cluster expands the osd, but the storage pool space becomes smaller

Hey David, In case this wasn't answered off list already: It looks like you have only added a single OSD to each new host? You specified 12*10T on osd{1..5}, and 12*12T on osd{6,7}. Just as a word of caution, the added 24T is more or less going to be wasted on osd{6,7} assuming that your crush

[ceph-users] Re: Howto upgrade AND change distro

I think it will depend on how you have your OSDs deployed currently. If they are bluestore deployed via ceph-volume using LVM, then it should mostly be pretty painless to migrate them to a new host, assuming everything is on the OSDs. The corner case would be if the WAL/DB is on a separate

[ceph-users] Re: Smarter DB disk replacement

I've been eyeing a similar icydock product (https://www.icydock.com/goods.php?id=309 ) for make M.2 drives more serviceable. While M.2 isn't ideal, if you have a 2U/4U box with a ton of available slots in the back, you could use these with some Micron

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

This is more the sentiment that I was hoping to convey. Sure, I have my finger on the pulse of the mailing list and the packages coming down the pipe, but assuming that everyone does and/or will is not a safe assumption. At the minimum, publishing the versioned repos at $repourl/debian-16.2.14

[ceph-users] Pacific 16.2.14 debian Incomplete

It looks like 16.2.14 was released, but it looks like in an incomplete way in the debian repo? I first noticed it because my nightly mirror snapshot picked it up, and showed that the majority of packages were removed, and only a handful had a new version. > focal-ceph-pacific 230829 to 230830

[ceph-users] rbd iostat requires pool specified

Hoping this may be trivial to point me towards, but I typically keep a background screen running `rbd perf image iostat` that shows all of the rbd devices with io, and how busy that disk may be at any given moment. Recently after upgrading everything to latest octopus release (15.2.16), it no

[ceph-users] Quincy 22.04/Jammy packages

Curious if there is a timeline for when quincy will start getting packages for Ubuntu Jammy/22.04. It looks like quincy started getting builds for EL9 with 17.2.4, and now with the 17.2.5 there are still only bullseye and focal dists available. Canonical is publishing a 17.2.0 build in

[ceph-users] Re: how to upgrade host os under ceph

You should be able to `do-release-upgrade` from bionic/18 to focal/20. Octopus/15 is shipped for both dists from ceph. Its been a while since I did this, the release upgrader might disable the ceph repo, and uninstall the ceph* packages. However, the OSDs should still be there, re-enable the

[ceph-users] Balancer Distribution Help

Hoping someone can point me to possible tunables that could hopefully better tighten my OSD distribution. Cluster is currently > "ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus > (stable)": 307 With plans to begin moving to pacific before end of year, with a possible

[ceph-users] Convert mon kv backend to rocksdb

Recently reading this thread: https://www.mail-archive.com/ceph-users@ceph.io/msg16705.html And out of curiosity I decided to take a look, and it turns out, 2/3 of my mons are using rocksdb, while I still somehow have a leveldb

[ceph-users] Re: linux distro requirements for reef

I will chime in just from my ubuntu perspective, if I compare previous (LTS) releases of ceph to ubuntu, there has typically been a 2 release cadence per ubuntu release. version U14 U16 U18 U20 U22 U24 jewel X X luminous X X mimic X X nautilous X X octopus X X pacific X X

[ceph-users] Re: MDS stuck ops

e is boosted. We are also on octopus. > > Best regards, > = > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > > From: Reed Dier mailto:reed.d...@focusvq.com>> > Sent: 28 November 2022 19:14:55

[ceph-users] Re: MDS stuck ops

gain, appreciate the pointers. Thanks, Reed > On Nov 28, 2022, at 11:41 AM, Venky Shankar wrote: > > On Mon, Nov 28, 2022 at 10:19 PM Reed Dier <mailto:reed.d...@focusvq.com>> wrote: >> >> Hopefully someone will be able to point me in the right direction h

[ceph-users] MDS stuck ops

Hopefully someone will be able to point me in the right direction here: Cluster is Octopus/15.2.17 on Ubuntu 20.04. All are kernel cephfs clients, either 5.4.0-131-generic or 5.15.0-52-generic. Cluster is nearful, and more storage is coming, but still 2-4 weeks out from delivery. > HEALTH_WARN

[ceph-users] Re: Permanently ignore some warning classes

You should be able to do `ceph health mute OSD_SLOW_PING_TIME_BACK --sticky` to mute the health warn/error state as it flaps. You can also set a TTL for the mute (1d, 1w, 1m) to have it roll off after a specific time. Code here is the warning error such as OSD_SLOW_PING_TIME_BACK or

[ceph-users] Re: Ceph OSD imbalance and performance

I think a few other things that could help would be `ceph osd df tree` which will show the hierarchy across different crush domains. And if you’re doing something like erasure coded pools, or something other than replication 3, maybe `ceph osd crush rule dump` may provide some further context

[ceph-users] Re: Ceph OSD imbalance and performance

Hope thats helpful, Reed > On Feb 28, 2023, at 1:11 PM, Dave Ingram wrote: > > > On Tue, Feb 28, 2023 at 12:56 PM Reed Dier <mailto:reed.d...@focusvq.com>> wrote: > I think a few other things that could help would be `ceph osd df tree` which > will show th

[ceph-users] Re: avg apply latency went up after update from octopus to pacific

Hi Jan, As someone who has been watching this thread in anticipation of planning an Octopus to Pacific upgrade, and also someone not all that interested in repaving all OSDs, which release(s) were the OSDs originally deployed with? Just trying to get a basic estimate on how recent or not these

[ceph-users] Pacific bluestore_volume_selection_policy

After ~3 uneventful weeks after upgrading from 15.2.17 to 16.2.14 I’ve started seeing OSD crashes with "cur >= fnode.size” and "cur >= p.length”, which seems to be resolved in the next point release for pacific later this month, but until then, I’d love to keep the OSDs from flapping. > $ for

[ceph-users] Re: Pacific bluestore_volume_selection_policy

;: "osd.12", So, I guess that configuration item doesn’t in fact prevent the crash as was purported. Looks like I may need to fast track moving to quincy… Reed > On Jan 8, 2024, at 9:47 AM, Reed Dier wrote: > > I ended up setting it in ceph.conf which appears to have worked (as far as

[ceph-users] Re: Pacific bluestore_volume_selection_policy

t; 3: abort() > 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x1b0) [0x56347eb33bec] > 5: /usr/bin/ceph-osd(+0x5d5daf) [0x56347eb33daf] > 6: (RocksDBBlueFSVolumeSelector::add_usage(void*, bluefs_fnode_t const&)+0) > [0x56347f1f7d00] > 7: (

[ceph-users] Re: Pacific bluestore_volume_selection_policy

ied to set it for one > OSD only and see if it starts with the config set? > > > Zitat von Reed Dier mailto:reed.d...@focusvq.com>>: > >> After ~3 uneventful weeks after upgrading from 15.2.17 to 16.2.14 I’ve >> started seeing OSD crashes with "cur >= fnode.siz

[ceph-users] Dropping focal for squid

Found this mention in the CLT Minutes posted this morning[1], of a discussion on ceph-dev[2] about dropping ubuntu focal builds for the squid release, and beginning builds of quincy for jammy to facilitate quincy->squid upgrades. > there was a consensus to drop support for ubuntu focal and

[ceph-users] Re: debian-reef_OLD?

Given that both the debian and rpm paths have been appended with _OLD, and this more recent post about 18.2.2 (hot-fix), it sounds like there is some sort of issue with 18.2.1? https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LEYDHWAPZW7KOGH2OH4TOPVGAFMZPYYP/