Hi,
take a look into 'ceph osd df' (maybe share the output) to see which
OSD(s) are full, they determine if when pool becomes full.
Did you delete lots of objects from that pool recently? That can take
some time until the space is really cleared.
Zitat von "Szabo, Istvan (Agoda)" :
Hi,
Here's what I learned about PG maps from my investigation of the code.
First, they don't seem to be involved in deciding what needs reconstruction
when a dead OSD is revived. There is a version number stored with the PGs
that is probably used for that.
It looks like nothing but statistics - the
This happens (unfortunately) frequently to me. Look for the active mgr
(ceph -s), and go restart the mgr service there (systemctl list-units |grep
mgr then systemctl restart NAMEOFSERVICE). This normally resolves that
error for me. You can look at the journalctl output and you'll likely see
errors
Hi,
I'm using Nautilus and I'm using the whole cluster mainly for a single
bucket in RadosGW.
There is a lot of data in this bucket (Petabyte scale) and I don't want to
waste all of SSD on it.
Is there anyway to automatically set some aging threshold for this data and
e.g. move any data older
Hi again,
One more update:
I connected to server-2 and ran ceph -s there. I got:
Error initializing cluster client: ObjectNotFound('RADOS object not found
(error calling conf_read_file)')
Today I created a RBD pool and created 2 RDB images in this pool. Could this be
reason for all
Hi,
I was browsing dashboard today. Then suddently it stopped working and i got 502
errors. I checked via root login and see thet ceph health is down to WARN.
I can access all rdb devices and CephFS. They work. All OSDs in server-1 is up.
health: HEALTH_WARN
1 hosts fail
Hello Zac,
I have some further questions on that page:
Right before the section "Delete Default Zone Group and Zone" there is another
warning that says:
"The following steps assume a multi-site configuration using newly installed
systems that aren’t storing data yet. DO NOT DELETE the
Hello Everyone,
I have installed both Prometheus and Grafana on one of my manager nodes (Ubuntu
18.04), and have configured both according to the documentation. I have visible
Grafana dashboards when visiting http://mon1:3000, but no data exists on the
dashboard. Python errors are shown for
Hi Andreas,
the cluster map and crush map are not the same thing. If you change the crush
map while the cluster is in degraded state, you basically modify this history
of cluster maps explicitly and have to live with the consequences (keeping
history under crush map changes is limited to up+in
You need a third monitor in order to form a quorum if one of the two
sites goes down. With only two sites, there is no safe way for them to
decide who is down.
On Tue, May 19, 2020 at 3:11 PM CodingSpiderFox
wrote:
>
> What does tiebreaker monitor mean? What exactly is its purpose?
>
What does tiebreaker monitor mean? What exactly is its purpose?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
Zac, can you confirm that this assumption is true?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
Great, thanks already.
I will study the publications of the project :)
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
Greg,
My name's Zac and I'm the docs guy for the Ceph Foundation. I have a
long-term plan to create a document that collects error codes and failure
cases, but I am only one man and it will be a few months before I can begin
on it.
Zac Dover
Ceph Docs Guy
On Wed, May 20, 2020 at 4:32 AM Gregory
On Tue, May 19, 2020 at 10:34 AM Benjeman Meekhof wrote:
>
> It is possible to run a ceph cluster over a WAN if you have reliable
> enough WAN with sites close enough for low-ish latency. The OSiRIS
> project is architected that way with Ceph services spread evenly
> across three university
I have been running Ceph over a gigabit WAN for a few months now and have been
happy with it. Mine is set up with Strongswan tunnels And dynamic routing with
BIRD) (although I would have used transport Mode and iBGP in hindsight). I
generally have 300-500kbps flow with 5ms latency.
What I
Hi Frank,
My understanding was that once a cluster is in a degraded state (an OSD
is down), ceph stores all changed cluster maps until the cluster is
healthy again exactly for the reason of finding missing objects. If
there is a real disaster of some kind, and many OSDs go up and down at
It is possible to run a ceph cluster over a WAN if you have reliable
enough WAN with sites close enough for low-ish latency. The OSiRIS
project is architected that way with Ceph services spread evenly
across three university sites in Michigan. There's more information
and contact on their
It is my understanding that it refers to running a single, normal ceph
cluster with it's component hosts connected over WAN. This would
require OSDs to connect to other OSDs and mons over WAN for nearly
every operation, and is not likely to perform acceptably.
On Tue, May 12, 2020 at 6:03 AM Wido den Hollander wrote:
> And to add to this: No, a newly created RBD image will never have 'left
> over' bits and bytes from a previous RBD image.
>
> I had to explain this multiple times to people which were used to old
> (i)SCSI setups where partitions could
The updated images have not been pushed to Dockerhub yet. I ran into the
same problem yesterday trying to update. Hopefully updated images will be
pushed on release (at the same time as the tarball release/prior to
announcement) moving forward in order to avoid this issue.
See here for latest
Hello everyone,
I'd like to setup a multisite ceph cluster.
Are there any sample setups that you can recommend studying?
I want to achieve fault tolerance but also I want to avoid split brain
scenarios.
I'm not that familiar with systems like ceph, so I would consider myself as
a beginner.
hi igor -
On 5/19/20 3:23 PM, Igor Fedotov wrote:
> slow_used_bytes is non-zero hence you have a spillover.
you are absolutely right, we do have spillovers on a large number of
osds. ceph tell osd.* compact is running right now.
> Additionally your DB volume size selection isn't perfect. For
Thoralf,
from your perf counter's dump:
"db_total_bytes": 15032377344,
"db_used_bytes": 411033600,
"wal_total_bytes": 0,
"wal_used_bytes": 0,
"slow_total_bytes": 94737203200,
"slow_used_bytes": 10714480640,
slow_used_bytes is non-zero hence you
On Tue, May 19, 2020 at 3:11 PM thoralf schulze
wrote:
>
> On 5/19/20 2:13 PM, Paul Emmerich wrote:
> > 3) if necessary add more OSDs; common problem is having very
> > few dedicated OSDs for the index pool; running the index on
> > all OSDs (and having a fast DB device for every disk) is
> >
hi igor, hi paul -
thank you for your answers.
On 5/19/20 2:05 PM, Igor Fedotov wrote:
> I presume that your OSDs suffer from slow RocksDB access,
> collection_listing operation is a culprit in this case - 30 items
> listing takes 96seconds to complete.
> From my experience such issues tend to
On Tue, May 19, 2020 at 2:06 PM Igor Fedotov wrote:
> Hi Thoralf,
>
> given the following indication from your logs:
>
> May 18 21:12:34 ceph-osd-05 ceph-osd[2356578]: 2020-05-18 21:12:34.211
> 7fb25cc80700 0 bluestore(/var/lib/ceph/osd/ceph-293) log_latency_fn
> slow operation observed for
Hi Thoralf,
given the following indication from your logs:
May 18 21:12:34 ceph-osd-05 ceph-osd[2356578]: 2020-05-18 21:12:34.211
7fb25cc80700 0 bluestore(/var/lib/ceph/osd/ceph-293) log_latency_fn
slow operation observed for _collection_list, latency = 96.337s, lat =
96s cid =2.0s2_head start
Hi Andreas,
I made exactly the same observation in another scenario. I added some OSDs
while other OSDs were down.
This is expected.
The crush map is an a-priory algorithm to compute the location of objects
without contacting a central server. Hence, *any*change of a crush map while an
OSD
Hi,
I have the following Ceph Mimic setup :
- a bunch of old servers with 3-4 SATA drives each (74 OSDs in total)
- index/leveldb is stored on each OSD (so no SSD drives, just SATA)
- the current usage is :
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
542 TiB 105 TiB
hi there,
we are seeing osd occasionally getting kicked out of our cluster, after
having been marked down by other osds. most of the time, the affected
osd rejoins the cluster after about ~5 minutes, but sometimes this takes
much longer. during that time, the osd seems to run just fine.
this
That's not wrong, those are expected objects that contain information
about your rbd images. If you take a look into the rbd_directory
(while you have images in there) you'll find something like this:
host:~ $ rados -p pool listomapkeys rbd_directory
id_fe976bcfb968bf
id_ffc37728edbdab
Was that a typo and you mean you changed min_size to 1? I/O paus with
min_size 1 and size 2 is unexpected, can you share more details like
your crushmap and your osd tree?
Zitat von Amudhan P :
Behaviour is same even after setting min_size 2.
On Mon 18 May, 2020, 12:34 PM Eugen Block,
Behaviour is same even after setting min_size 2.
On Mon 18 May, 2020, 12:34 PM Eugen Block, wrote:
> If your pool has a min_size 2 and size 2 (always a bad idea) it will
> pause IO in case of a failure until the recovery has finished. So the
> described behaviour is expected.
>
>
> Zitat von
34 matches
Mail list logo
