The error will go away once you start storing data in the other pools. Or,
you could simply silence the message with mon_pg_warn_max_object_skew = 0
On Thu, Jun 7, 2018 at 10:48 AM, Torin Woltjer
wrote:
> I have a ceph cluster and status shows this error: pool libvirt-pool has
> many more
Hi all!
I'm having trouble creating OSDs on some boxes that once held Bluestore
OSDs. I have rolled the ceph software back from 12.2.4 -> 10.2.9 on the
boxes, but I'm running into this error when creating osds.
2018-06-12 22:32:42.78 7fcaf39e2800 0 ceph version 10.2.9
Hi All,
I'm having issues trying to get a 2nd Rados GW realm/zone up and
running. The configuration seemed to go well, but I'm unable to start the
gateway.
2018-05-29 21:21:27.119192 7fd26cfdd9c0 0 ERROR: failed to decode obj from
.rgw.root:zone_info.fe2e0680-d7e8-415f-bf91-501dda96d075
_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": f
Hi All. I have a ceph cluster that's partially upgraded to Luminous. Last
night a host died and since then the cluster is failing to recover. It
finished backfilling, but was left with thousands of requests degraded,
inactive, or stale. In order to move past the issue, I put the cluster in
Hi Cephers,
I am in the process of upgrading a cluster from Filestore to bluestore,
but I'm concerned about frequent warnings popping up against the new
bluestore devices. I'm frequently seeing messages like this, although the
specific osd changes, it's always one of the few hosts I've converted
* the system is healthy, no swapping, no high load, no errors in dmesg
>
> I attached a log excerpt of osd.35 - probably this is useful for
> investigating the problem is someone owns deeper bluestore knowledge.
> (slow requests appeared on Sun Sep 2 21:00:35)
>
> Regards
> Mar
un? Would you be able to test against the same kernel
> with Spectre/Meltdown mitigations disabled ("noibrs noibpb nopti
> nospectre_v2" as boot option)?
>
> Uwe
>
> Am 05.09.18 um 19:30 schrieb Brett Chancellor:
>
>> Marc,
>>As with you, this
0.9
>> bluestore cache meta ratio = 0.1
>> bluestore cache size hdd = 10737418240
>> * this type problem only appears on hdd/bluestore osds, ssd/bluestore
>> osds did never experienced that problem
>> * the system is healthy, no swapping, no high
I saw above the recommended size for the db partition was 5% of data, but
yet the recommendation is 40GB partitions for 4TB drives. Isn't that closer
to 1%?
On Fri, Sep 7, 2018 at 10:06 AM, Muhammad Junaid
wrote:
> Thanks very much. It is clear very much now. Because we are just in
> planning
The warnings look like this.
6 ops are blocked > 32.768 sec on osd.219
1 osds have slow requests
On Sun, Sep 2, 2018, 8:45 AM Alfredo Deza wrote:
> On Sat, Sep 1, 2018 at 12:45 PM, Brett Chancellor
> wrote:
> > Hi Cephers,
> > I am in the process of upgrading a c
.
> Scrubs might don't want to start if there are many other scrubs ongoing.
>
> czw., 11 paź 2018 o 14:27 Brett Chancellor
> napisał(a):
>
>> I moved the file. But the cluster won't actually start any scrub/repair I
>> manually initiate.
>>
>> On Thu, Oct
art
> is over the deep scrub continues until it is done.
>
> On Thu, Oct 11, 2018, 12:26 PM Brett Chancellor <
> bchancel...@salesforce.com> wrote:
>
>> Does the "repair" function use the same rules as a deep scrub? I couldn't
>> get one to kick off
You could also set *osd_crush_initial_weight = 0 . *New OSDs will
automatically come up with a 0 weight and you won't have to race the clock.
-Brett
On Thu, Oct 4, 2018 at 3:50 AM Janne Johansson wrote:
>
>
> Den tors 4 okt. 2018 kl 00:09 skrev Bruno Carvalho :
>
>> Hi Cephers, I would like to
an OSD.
>>
>> There is such less information about this tools. So it might be risky. I
>> hope someone much experienced could help more.
>>
>>
>> > On 2 Oct 2018, at 23:23, Brett Chancellor
>> wrote:
>> >
>> > Help. I have a 60 node clust
Help. I have a 60 node cluster and most of the OSDs decided to crash
themselves at the same time. They wont restart, the messages look like...
--- begin dump of recent events ---
0> 2018-10-02 21:19:16.990369 7f57ab5b7d80 -1 *** Caught signal
(Aborted) **
in thread 7f57ab5b7d80
Hi all,
I have an inconsistent PG. I've tried running a repair and manual deep
scrub, but neither operation seems to actually do anything. I've also
tried stopping the primary OSD, removing the object, and restarting the
OSD. The system copies the object back, but the inconsistent PG ERR
Hi all. Quick question about osd metadata information. I have several OSDs
setup with the data dir on HDD and the db going to a partition on ssd. But
when I look at the metadata for all the OSDs, it's showing the db as "hdd".
Does this effect anything? And is there anyway to change it?
$ sudo
This seems right. You are doing a single benchmark from a single client.
Your limiting factor will be the network latency. For most networks this is
between 0.2 and 0.3ms. if you're trying to test the potential of your
cluster, you'll need multiple workers and clients.
On Thu, Feb 7, 2019, 2:17
After upgrading from 14.2.0 to 14.2.1, I've noticed PGs are frequently
resetting their scrub and deep scrub time stamps to 0.00. It's extra
strange because the peers show timestamps for deep scrubs.
## First entry from a pg list at 7pm
$ grep 11.2f2 ~/pgs-active.7pm
11.2f2 6910
Not sure if it's related, but this only happens to PG's who's primary OSD
is one where osd_numa_node has been set.
On Wed, May 15, 2019 at 7:13 PM Brett Chancellor
wrote:
> After upgrading from 14.2.0 to 14.2.1, I've noticed PGs are frequently
> resetting their scrub and deep scrub time
It works okay. You need a ceph.conf and a generic radosgw cephx key. That's
it.
On Wed, Jun 5, 2019, 5:37 AM Marc Roos wrote:
>
>
> Has anyone put the radosgw in a container? What files do I need to put
> in the sandbox directory? Are there other things I should consider?
>
>
>
>
If you don't figure out how to enable it on your monitor, you can always
disable it to squash the warnings
*ceph config set mon.node01 ms_bind_msgr2 false*
On Fri, Jun 14, 2019 at 12:11 PM Bob Farrell wrote:
> Hi. Firstly thanks to all involved in this great mailing list, I learn
> lots from it
radosgw will try and create all if the default pools if they are missing.
The number of pools changes depending on the version, but it's somewhere
around 5.
On Sun, Jun 9, 2019, 1:00 PM wrote:
> Huan;
>
> I get that, but the pool already exists, why is radosgw trying to create
> one?
>
>
Does anybody have a fix for BlueFS spillover detected? This started
happening 2 days after an upgrade to 14.2.1 and has increased from 3 OSDs
to 118 in the last 4 days. I read you could fix it by rebuilding the OSDs,
but rebuilding the 264 OSDs on this cluster will take months of rebalancing.
$
is kept at slow device - which might has some negative performance impact.
>
> Unfortunately that's a know issue with current RocksDB/BlueStore
> interaction - spillovers to slow device might take place even when there is
> plenty of free space at fast one.
>
>
> Thanks,
>
>
Background: We have a few ceph clusters, each serves multiple Openstack
cluster. Each cluster has it's own set of pools.
I'd like to move ~50TB of volumes from an old cluster (we'll call the pool
cluster01-volumes) to an existing pool (cluster02-volumes) to later be
imported by a different
Both pools are in the same Ceph cluster. Do you have any documentation on
the live migration process? I'm running 14.2.1
On Wed, Jun 19, 2019, 8:35 PM Jason Dillaman wrote:
> On Wed, Jun 19, 2019 at 6:25 PM Brett Chancellor
> wrote:
> >
> > Background: We have a few ceph clu
Has anybody else encountered this issue? Prometheus is failing to scrape
the prometheus module, returning invalid metric type
"cef431ab_b67a_43f9_9b87_ebe992dec94e_replay_bytes counter"
Ceph version: 14.2.1
Prometheus version: 2.10.0-rc.0
This started happening when I setup one way rbd mirroring
I have used the gentle reweight script many times in the past. But more
recently, I expanded one cluster from 334 to 1114 OSDs, by just changing
the crush weight 100 OSDs at a time. Once all pgs from those 100 were
stable and backfilling, add another hundred. I stopped at 500 and let the
backfill
You can increase your scrub intervals.
osd deep scrub interval
osd scrub max interval
On Tue, May 14, 2019 at 7:00 AM EDH - Manuel Rios Fernandez <
mrios...@easydatahost.com> wrote:
> Hi Muthu
>
>
>
> We found the same issue near 2000 pgs not deep-scrubbed in time.
>
>
>
> We’re manually force
I would try the ceph pg repair. If you see the pg go into deep scrubbing,
then back to inconsistent you probably have a bad drive. Find which of the
drives in the pg are bad (pg query or go to the host and look through
dmesg). Take that osd offline and mark it out. Once backfill is complete,
it
, 2019, 6:41 PM Jorge Garcia wrote:
> I have tried ceph pg repair several times. It claims "instructing pg
> 2.798s0 on osd.41 to repair" but then nothing happens as far as I can tell.
> Any way of knowing if it's doing more?
>
> On Sat, May 18, 2019 at 3:33 PM Bret
t: Re: [ceph-users] 3 OSDs stopped and unable to restart
> From: Brett Chancellor
> To: Igor Fedotov
> CC: Ceph Users
>
> Once backfilling finished, the cluster was super slow, most osd's were
> filled with heartbeat_map errors. When an OSD restarts it causes a cascade
> of oth
> At least at the moment.
>
> In short the history is: starting OSD tries to flush bluefs data to disk,
> detects lack of space and asks for more from main device - allocations
> succeeds but returned extent has length field set to 0.
> On 7/9/2019 8:33 PM, Brett Chancellor
I did try and run sudo ceph-bluestore-tool --out-dir /mnt/ceph
bluefs-export . but it died after writing out 93GB and filling up my root
partition.
On Thu, Jul 11, 2019 at 3:32 PM Brett Chancellor
wrote:
> We moved the .rgw.meta data pool over to SSD to try and improve
> performance,
some posts about that behavior in the mail list this
> year.
>
> Thanks,
>
> Igor.
>
>
> On 7/8/2019 8:33 PM, Brett Chancellor wrote:
>
>
> I'll give that a try. Is it something like...
> ceph tell 'osd.*' bluestore_allocator stupid
> ceph tell 'osd.*' bluefs_allocat
Hi All! Today I've had 3 OSDs stop themselves and are unable to restart,
all with the same error. These OSDs are all on different hosts. All are
running 14.2.1
I did try the following two commands
- ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-80 list > keys
## This failed with the
Ceph already does this by default. For each replicated pool, you can set
the 'size' which is the number of copies you want Ceph to maintain. The
accepted norm for replicas is 3, but you can set it higher if you want to
incur the performance penalty.
On Mon, Jul 1, 2019, 6:01 AM nokia ceph wrote:
7fc732475700 5
bluestore(/var/lib/ceph/osd/ceph-59) allocate_bluefs_freespace gifting
0x4d18d0~40 to bluefs
Any thoughts?
On Sat, Jul 6, 2019 at 3:06 PM Brett Chancellor
wrote:
> Has anybody else run into this? It seems to be slowly spreading to other
> OSDs, maybe it gets to a
ceph-bluestore-tool --path path-to-osd!!!> bluefs-bdev-sizes" and share the output.
>
> Thanks,
>
> Igor
> On 7/3/2019 9:59 PM, Brett Chancellor wrote:
>
> Hi All! Today I've had 3 OSDs stop themselves and are unable to restart,
> all with the same error. These OSDs
In Nautilus just pg_num is sufficient for both increases and decreases.
On Mon, Jul 1, 2019 at 10:55 AM Robert LeBlanc wrote:
> I believe he needs to increase the pgp_num first, then pg_num.
>
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
If all you want to do is repair the pg when it finds an inconsistent pg,
you could set osd_scrub_auto_repair to true.
On Sun, Aug 4, 2019, 9:16 AM Harry G. Coin wrote:
> Question: If you have enough osds it seems an almost daily thing when
> you get to work in the morning there' s a "ceph
y placement groups for the SSDs, but also I don't
> know that there were that many objects. There were of course a ton of
> omap key/values.
>
> -Troy
>
> On 8/18/19 10:57 PM, Brett Chancellor wrote:
> > This sounds familiar. Do any of these pools on the SSD have fairly dense
This sounds familiar. Do any of these pools on the SSD have fairly dense
placement group to object ratios? Like more than 500k objects per pg? (ceph
pg ls)
On Sun, Aug 18, 2019, 10:12 PM Brad Hubbard wrote:
> On Thu, Aug 15, 2019 at 2:09 AM Troy Ablan wrote:
> >
> > Paul,
> >
> > Thanks for
clearing it out? I don't see any other references to
large omap in any of the logs. I've tried restarted the mgr's, the
monitors, and even the osd that reported the issue.
-Brett
On Thu, Jul 25, 2019 at 2:55 PM Brett Chancellor
wrote:
> 14.2.1
> Thanks, I'll try that.
>
> On Thu,
I was able to answer my own question. For future interested parties, I
initiated a deep scrub on the placement group, which cleared the error.
On Tue, Jul 30, 2019 at 1:48 PM Brett Chancellor
wrote:
> I was able to remove the meta objects, but the cluster is still in WARN
> state
> HE
I'm having an issue similar to
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-March/033611.html .
I don't see where any solution was proposed.
$ ceph health detail
HEALTH_WARN 1 large omap objects
LARGE_OMAP_OBJECTS 1 large omap objects
1 large objects found in pool
dmin sync
> status' on any zone show that it's stuck behind on metadata sync? That
> would prevent these logs from being trimmed and result in these large
> omap warnings.
>
> On 7/25/19 1:59 PM, Brett Chancellor wrote:
> > I'm having an issue similar to
> >
> http://l
ow that it's stuck behind on metadata sync?
> > That
> > would prevent these logs from being trimmed and result in these large
> > omap warnings.
> >
> > On 7/25/19 1:59 PM, Brett Chancellor wrote:
> > > I'm having an issue similar to
runs out of space.
On Wed, Jul 3, 2019 at 2:59 PM Brett Chancellor
wrote:
> Hi All! Today I've had 3 OSDs stop themselves and are unable to restart,
> all with the same error. These OSDs are all on different hosts. All are
> running 14.2.1
>
> I did try the following two commands
&
, Jul 9, 2019 at 1:03 PM Igor Fedotov wrote:
> Please try to set bluestore_bluefs_gift_ratio to 0.0002
>
>
> On 7/9/2019 7:39 PM, Brett Chancellor wrote:
>
> Too large for pastebin.. The problem is continually crashing new OSDs.
> Here is the latest one.
>
> On Tue, J
52 matches
Mail list logo
