Hello Eugen,
before answering your questions:
Creating a new simple replicated crush rule and setting it to the
metadata pool was the solution.
> - Your cache tier was on SSDs which need to be removed.
Yes, the cache tier pool and the metadata pool for the RBDs.
> - Cache tier was removed
Hi,
So now we need to empty these OSDs.
The device class was SSD. I changed it to HDD and moved the OSDs
inside the Crush tree to the other HDD OSDs of the host.
I need to move the PGs away from the OSDs to other OSDs but I do not
know how to do it.
your crush rule doesn't specify a
Hello,
well yes, I think I have to edit the Crush rule and modify:
item_name
or to be clear:
I need to modify this in the decompiled crush map:
root bmeta {
id -4 # do not change unnecessarily
id -254 class hdd # do not change unnecessarily
id
Hello Eugen,
I was wrong. I am sorry.
The PGs are not empty and orphaned.
Most of the PGs are empty but a few are indeed used.
And the pool for these PGs is still there. It is the metadata pool of
the erasure coded pool for RBDs. The cache tier pool was removed
successfully.
So now we
@Eugen
We have seen the same problems 8 years ago. I can only recommend never
to use cache tiering in production.
At Cephalocon this was part of my talk and as far as I remember cache
tiering will also disappear from ceph soon.
Cache tiering has been deprecated in the Reef release as it has
Maybe the mismatching OSD versions had an impact on the unclean tier
removal, but this is just a guess. I couldn't reproduce it in a
Pacific test cluster, the removal worked fine without leaving behind
empty PGs. But I had only a few rbd images in that pool so it's not
really
Hello Eugen, Hello Joachim,
@Joachim: Interesting! And you got empty PGs, too? How did you solve the
problem?
@Eugen: This is one of our biggest clusters and we're in the process to
migrate from Nautilus to Octopus and to migrate from CentOS to Ubuntu.
The cache tier pool's OSDs were still
I know, I know... but since we are already using it (for years) I have
to check how to remove it safely, maybe as long as we're on Pacific. ;-)
Zitat von Joachim Kraftmayer - ceph ambassador :
@Eugen
We have seen the same problems 8 years ago. I can only recommend
never to use cache
@Eugen
We have seen the same problems 8 years ago. I can only recommend never
to use cache tiering in production.
At Cephalocon this was part of my talk and as far as I remember cache
tiering will also disappear from ceph soon.
Cache tiering has been deprecated in the Reef release as it has
Which ceph version is this? I'm trying to understand how removing a
pool leaves the PGs of that pool... Do you have any logs or something
from when you removed the pool?
We'll have to deal with a cache tier in the forseeable future as well
so this is quite relevant for us as well. Maybe I'll
Hello Eugen,
yes, we followed the documentation and everything worked fine. The cache
is gone.
Removing the pool worked well. Everything is clean.
The PGs are empty active+clean.
Possible solutions:
1.
ceph pg {pg-id} mark_unfound_lost delete
I do not think this is the right way since it
Hi,
just for clarity, you're actually talking about the cache tier as
described in the docs [1]? And you followed the steps until 'ceph osd
tier remove cold-storage hot-storage' successfully? And the pool has
been really deleted successfully ('ceph osd pool ls detail')?
[1]
12 matches
Mail list logo
