[ceph-users] Too few PGs per OSD (autoscaler)

Helo, Ceph users, TL;DR: PG autoscaler should not cause the "too few PGs per OSD" warning Detailed: Some time ago, I upgraded the HW in my virtualization+Ceph cluster, replacing 30+ old servers with <10 modern servers. I immediately got "Too much PGs per OSD" warning, so I had to add more

Re: [ceph-users] How do you deal with "clock skew detected"?

Konstantin Shalygin wrote: : >how do you deal with the "clock skew detected" HEALTH_WARN message? : > : >I think the internal RTC in most x86 servers does have 1 second resolution : >only, but Ceph skew limit is much smaller than that. So every time I reboot : >one of my mons (for kernel upgrade or

[ceph-users] Huge rebalance after rebooting OSD host (Mimic)

Hello, Ceph users, I wanted to install the recent kernel update on my OSD hosts with CentOS 7, Ceph 13.2.5 Mimic. So I set a noout flag and ran "yum -y update" on the first OSD host. This host has 8 bluestore OSDs with data on HDDs and database on LVs of two SSDs (each SSD has 4 LVs for OS

[ceph-users] How do you deal with "clock skew detected"?

Hello, Ceph users, how do you deal with the "clock skew detected" HEALTH_WARN message? I think the internal RTC in most x86 servers does have 1 second resolution only, but Ceph skew limit is much smaller than that. So every time I reboot one of my mons (for kernel upgrade or something), I

Re: [ceph-users] Radosgw object size limit?

Hello, thanks for your help. Casey Bodley wrote: : It looks like the default.rgw.buckets.non-ec pool is missing, which : is where we track in-progress multipart uploads. So I'm guessing : that your perl client is not doing a multipart upload, where s3cmd : does by default. : : I'd recomm

Re: [ceph-users] Radosgw object size limit?

which pool or other parameter is the problem? I have the following pools: # ceph osd pool ls one .rgw.root default.rgw.control default.rgw.meta default.rgw.log default.rgw.buckets.index default.rgw.buckets.data (the "one" pool is unrelated to RadosGW, it contains OpenNebula RBD images). Thanks, -Yenya : On

[ceph-users] Radosgw object size limit?

Hello, Ceph users, does radosgw have an upper limit of object size? I tried to upload a 11GB file using s3cmd, but it failed with InvalidRange error: $ s3cmd put --verbose centos/7/isos/x86_64/CentOS-7-x86_64-Everything-1810.iso s3://mybucket/ INFO: No cache file found, creating it. INFO

Re: [ceph-users] RBD image format v1 EOL ...

Hello, Jason Dillaman wrote: : For the future Ceph Octopus release, I would like to remove all : remaining support for RBD image format v1 images baring any : substantial pushback. : : The image format for new images has been defaulted to the v2 image : format since Infernalis, the v1 for

Re: [ceph-users] Bluestore increased disk usage

. Sincerely, -Yenya : pt., 8 lut 2019, 10:11: Jan Kasprzak napisał(a): : : > Hello, ceph users, : > : > I moved my cluster to bluestore (Ceph Mimic), and now I see the increased : > disk usage. From ceph -s: : > : > pools: 8 pools, 3328 pgs : > objects

[ceph-users] Bluestore increased disk usage

Hello, ceph users, I moved my cluster to bluestore (Ceph Mimic), and now I see the increased disk usage. From ceph -s: pools: 8 pools, 3328 pgs objects: 1.23 M objects, 4.6 TiB usage: 23 TiB used, 444 TiB / 467 TiB avail I use 3-way replication of my data, so I would expe

Re: [ceph-users] Downsizing a cephfs pool

Hello, Brian Topping wrote: : Hi all, I created a problem when moving data to Ceph and I would be grateful for some guidance before I do something dumb. [...] : Do I need to create new pools and copy again using cpio? Is there a better way? I think I will be facing the same prob

[ceph-users] pgs inactive after setting a new crush rule (Re: backfill_toofull after adding new OSDs)

Jan Kasprzak wrote: : OKay, now I changed the crush rule also on a pool with : the real data, and it seems all the client i/o on that pool has stopped. : The recovery continues, but things like qemu I/O, "rbd ls", and so on : are just stuck doing nothing. : : Can I unstuck

Re: [ceph-users] backfill_toofull after adding new OSDs

s -Yenya Jan Kasprzak wrote: : : - Original Message - : : From: "Caspar Smit" : : To: "Jan Kasprzak" : : Cc: "ceph-users" : : Sent: Thursday, 31 January, 2019 15:43:07 : : Subject: Re: [ceph-users] backfill_toofull after adding new OSDs : : : : Hi Jan,

Re: [ceph-users] backfill_toofull after adding new OSDs

g the data reshuffle. 13.2.4 on CentOS 7. -Yenya : : - Original Message - : From: "Caspar Smit" : To: "Jan Kasprzak" : Cc: "ceph-users" : Sent: Thursday, 31 January, 2019 15:43:07 : Subject: Re: [ceph-users] backfill_toofull after adding new OSDs : :

[ceph-users] backfill_toofull after adding new OSDs

Hello, ceph users, I see the following HEALTH_ERR during cluster rebalance: Degraded data redundancy (low space): 8 pgs backfill_toofull Detailed description: I have upgraded my cluster to mimic and added 16 new bluestore OSDs on 4 hosts. The hosts are in a separate region in my

Re: [ceph-users] Spec for Ceph Mon+Mgr?

jes...@krogh.cc wrote: : Hi. : : We're currently co-locating our mons with the head node of our Hadoop : installation. That may be giving us some problems, we dont know yet, but : thus I'm speculation about moving them to dedicated hardware. : : It is hard to get specifications "small" engough ..

Re: [ceph-users] Migrating to a dedicated cluster network

one network for next cluster setup. -Yenya : śr., 23 sty 2019, 10:40: Jan Kasprzak napisał(a): : : > Hello, Ceph users, : > : > is it possible to migrate already deployed Ceph cluster, which uses : > public network only, to a split public/dedicated networks? If so, : &g

[ceph-users] Migrating to a dedicated cluster network

Hello, Ceph users, is it possible to migrate already deployed Ceph cluster, which uses public network only, to a split public/dedicated networks? If so, can this be done without service disruption? I have now got a new hardware which makes this possible, but I am not sure how to do it.

Re: [ceph-users] block.db on a LV? (Re: Mixed SSD+HDD OSD setup recommendation)

Alfredo, Alfredo Deza wrote: : On Fri, Jan 18, 2019 at 7:21 AM Jan Kasprzak wrote: : > Eugen Block wrote: : > : : > : I think you're running into an issue reported a couple of times. : > : For the use of LVM you have to specify the name of the Volume Group : >

Re: [ceph-users] block.db on a LV? (Re: Mixed SSD+HDD OSD setup recommendation)

Eugen Block wrote: : Hi Jan, : : I think you're running into an issue reported a couple of times. : For the use of LVM you have to specify the name of the Volume Group : and the respective Logical Volume instead of the path, e.g. : : ceph-volume lvm prepare --bluestore --block.db ssd_vg/ssd00 --d

[ceph-users] block.db on a LV? (Re: Mixed SSD+HDD OSD setup recommendation)

Hello, Ceph users, replying to my own post from several weeks ago: Jan Kasprzak wrote: : [...] I plan to add new OSD hosts, : and I am looking for setup recommendations. : : Intended usage: : : - small-ish pool (tens of TB) for RBD volumes used by QEMU : - large pool for object-based

[ceph-users] Radosgw cannot create pool

its with the similar message: # radosgw-admin user create --uid=kas --display-name="Jan Kasprzak" 2019-01-17 09:52:29.805828 7fea6cfd2dc0 0 rgw_init_ioctx ERROR: librados::Rados::pool_create returned (34) Numerical result out of range (this can be due to a pool or placement group misconfi

[ceph-users] Get packages - incorrect link

Hello, Ceph users, I am not sure where to report the issue with the ceph.com website, so I am posting to this list: The https://ceph.com/use/ page has an incorrect link for getting the packages: "For packages, see http://ceph.com/docs/master/install/get-packages"; - the URL should be ht

Re: [ceph-users] Ceph blog RSS/Atom URL?

/vdbench%e6%b5%8b%e8%af%95%e5%ae%9e%e6%97%b6%e5%8f%af%e8%a7%86%e5%8c%96%e6%98%be%e7%a4%ba/ -Yenya : On Fri, Jan 4, 2019 at 5:52 AM Jan Kasprzak wrote: : > is there any RSS or Atom source for Ceph blog? I have looked inside : > the https://ceph.com/community/blog/ HTML source, but there is no :

Re: [ceph-users] ceph health JSON format has changed

Gregory Farnum wrote: : On Wed, Jan 2, 2019 at 5:12 AM Jan Kasprzak wrote: : : > Thomas Byrne - UKRI STFC wrote: : > : I recently spent some time looking at this, I believe the 'summary' and : > : 'overall_status' sections are now deprecated. The 'status'

[ceph-users] Ceph blog RSS/Atom URL?

Hello, is there any RSS or Atom source for Ceph blog? I have looked inside the https://ceph.com/community/blog/ HTML source, but there is no or anything mentioning RSS or Atom. Thanks, -Yenya -- | Jan "Yenya" Kasprzak | | http://www.fi.muni.cz/~kas/ GPG: 4096R

Re: [ceph-users] ceph health JSON format has changed

Thomas Byrne - UKRI STFC wrote: : I recently spent some time looking at this, I believe the 'summary' and : 'overall_status' sections are now deprecated. The 'status' and 'checks' : fields are the ones to use now. OK, thanks. : The 'status' field gives you the OK/WARN/ERR, but returning t

[ceph-users] ceph health JSON format has changed sync?

Hello, Ceph users, I am afraid the following question is a FAQ, but I still was not able to find the answer: I use ceph --status --format=json-pretty as a source of CEPH status for my Nagios monitoring. After upgrading to Luminous, I see the following in the JSON output when the cluster i

[ceph-users] Mixed SSD+HDD OSD setup recommendation

Hello, CEPH users, having upgraded my CEPH cluster to Luminous, I plan to add new OSD hosts, and I am looking for setup recommendations. Intended usage: - small-ish pool (tens of TB) for RBD volumes used by QEMU - large pool for object-based cold (or not-so-hot :-) data, write-on

Re: [ceph-users] Upgrade to Luminous (mon+osd)

ster? Contact us at https://croit.io : > : > croit GmbH : > Freseniusstr. 31h : > 81247 München : > www.croit.io : > Tel: +49 89 1896585 90 : > : > Am Mo., 3. Dez. 2018 um 11:56 Uhr schrieb Jan Kasprzak : : > > : > > Hello, ceph users, : > > : > &g

Re: [ceph-users] Upgrade to Luminous (mon+osd)

l 2>&1 fi [...] So maybe ceph-selinux should also honor CEPH_AUTO_RESTART_ON_UPGRADE=no in /etc/sysconfig/ceph ? But I am not sure whether it is possible at all, when the labels got changed. -Yenya : Am Mo., 3. Dez. 2018 um 11:56 Uhr schrieb Jan Kasprzak : : > : > I have a small(-i

[ceph-users] Upgrade to Luminous (mon+osd)

Hello, ceph users, I have a small(-ish) Ceph cluster, where there are osds on each host, and in addition to that, there are mons on the first three hosts. Is it possible to upgrade the cluster to Luminous without service interruption? I have tested that when I run "yum --enablerepo Ceph u

[ceph-users] Atomic object replacement with libradosstriper

Hello, Ceph users, I would like to use RADOS as an object storage (I have written about it to this list a while ago), and I would like to use libradosstriper with C, as has been suggested to me here. My question is - when writing an object, is it possible to do it so that either t

Re: [ceph-users] pgs stuck unclean after removing OSDs

David Turner wrote: : A couple things. You didn't `ceph osd crush remove osd.21` after doing the : other bits. Also you will want to remove the bucket (re: host) from the : crush map as it will now be empty. Right now you have a host in the crush : map with a weight, but no osds to put that data

[ceph-users] pgs stuck unclean after removing OSDs

Hello, TL;DR: what to do when my cluster reports stuck unclean pgs? Detailed description: One of the nodes in my cluster died. CEPH correctly rebalanced itself, and reached the HEALTH_OK state. I have looked at the failed server, and decided to take it out of the cluster permanently, bec

Re: [ceph-users] rados rm: device or resource busy

Hello, Brad Hubbard wrote: : I can reproduce this. [...] : That's here where you will notice it is returning EBUSY which is error : code 16, "Device or resource busy". : : https://github.com/badone/ceph/blob/wip-ceph_test_admin_socket_output/src/cls/lock/cls_lock.cc#L189 : : In order t

Re: [ceph-users] rados rm: device or resource busy

er this is probably a : prime time to figure out how to get past this in case it happens in the : future in production. Yes. This is why I am asking now. -Yenya : On Thu, Jun 8, 2017 at 11:04 AM Jan Kasprzak wrote: : > I have created a RADOS striped object using : > : > $

[ceph-users] rados rm: device or resource busy

Hello, I have created a RADOS striped object using $ dd someargs | rados --pool testpool --striper put testfile - and interrupted it in the middle of writing. Now I cannot remove this object: $ rados --pool testpool --striper rm testfile error removing testpool>testfile: (16) Device or

Re: [ceph-users] RADOS as a simple object storage

Wido den Hollander wrote: : : > Op 27 februari 2017 om 15:59 schreef Jan Kasprzak : : > : > : > Here is some statistics from our biggest instance of the object storage: : > : > : > : > : > : > objects stored: 100_000_000 : > : > : > < 1024 byt

Re: [ceph-users] RADOS as a simple object storage

Hello, Gregory Farnum wrote: : On Mon, Feb 20, 2017 at 11:57 AM, Jan Kasprzak wrote: : > Gregory Farnum wrote: : > : On Mon, Feb 20, 2017 at 6:46 AM, Jan Kasprzak wrote: : > : > : > : > I have been using CEPH RBD for a year or so as a virtual machine storage : >

Re: [ceph-users] RADOS as a simple object storage

Gregory Farnum wrote: : On Mon, Feb 20, 2017 at 6:46 AM, Jan Kasprzak wrote: : > Hello, world!\n : > : > I have been using CEPH RBD for a year or so as a virtual machine storage : > backend, and I am thinking about moving our another subsystem to CEPH: : > : > The subsys

[ceph-users] RADOS as a simple object storage

Hello, world!\n I have been using CEPH RBD for a year or so as a virtual machine storage backend, and I am thinking about moving our another subsystem to CEPH: The subsystem in question is a simple replicated object storage, currently implemented by a custom C code by yours truly. My ques