Re: [ceph-users] ceph mgr balancer bad distribution

Does anybody have some more input? I keeped the balancer active for 24h now and it is rebalancing 1-3% every 30 minutes but the distribution is still bad. It seems to balance from left to right and than back from right to left... Greets, Stefan Am 28.02.2018 um 13:47 schrieb Stefan Priebe -

Re: [ceph-users] Ceph Developer Monthly - March 2018

On Wed, Feb 28, 2018 at 10:51:29PM +, Sage Weil wrote: > On Wed, 28 Feb 2018, Dan Mick wrote: > > Would anyone else appreciate a Google Calendar invitation for the CDMs? > > Seems like a natural. > > Funny you should mention it! I was just talking to Leo this morning about > creating a

Re: [ceph-users] ceph-deploy won't install luminous (but Jewel instead)

Hi Max, I had the same issue (under Ubuntu 16.04) but I have read the ceph-deploy 2.0.0 source code and saw a "—-release" flag for the install subcommand. You can found the flag with the following command: ceph-deploy install --help It looks like the culprit part of ceph-deploy can be found

Re: [ceph-users] ceph-deploy won't install luminous (but Jewel instead)

Didn't check at time. I deployed everything from VM standalone. The VM was just build up with fresh new centOS7.4 using minimal installation ISO1708. It's a completly new/fresh/empty system. Then I run: yum update -y yum install wget zip unzip vim pciutils -y yum install epel-release -y yum

Re: [ceph-users] Cannot Create MGR

Indeed that was the problem! In case anyone else ever comes to the same condition please keep in mind that no matter what you write at the "ceph-deploy" command it will use at some point the the output from "hostname -s" and try to connect to gather data from that monitor. If you have

Re: [ceph-users] Ceph Developer Monthly - March 2018

On Wed, 28 Feb 2018, Dan Mick wrote: > Would anyone else appreciate a Google Calendar invitation for the CDMs? > Seems like a natural. Funny you should mention it! I was just talking to Leo this morning about creating a public Ceph Events calendar that has all of the public events (CDM, tech

Re: [ceph-users] Ceph Developer Monthly - March 2018

Would anyone else appreciate a Google Calendar invitation for the CDMs? Seems like a natural. On 02/27/2018 09:37 PM, Leonardo Vaz wrote: > Hey Cephers, > > This is just a friendly reminder that the next Ceph Developer Monthly > meeting is coming up: > > http://wiki.ceph.com/Planning > > If

Re: [ceph-users] ceph-deploy won't install luminous (but Jewel instead)

Which version of ceph-deploy are you using? On Wed, Feb 28, 2018 at 4:37 AM Massimiliano Cuttini wrote: > This worked. > > However somebody should investigate why default is still jewel on Centos > 7.4 > > Il 28/02/2018 00:53, jorpilo ha scritto: > > Try using: > ceph-deploy

Re: [ceph-users] OSD Segfaults after Bluestore conversion

I'm following up from awhile ago. I don't think this is the same bug. The bug referenced shows "abort: Corruption: block checksum mismatch", and I'm not seeing that on mine. Now I've had 8 OSDs down on this one server for a couple of weeks, and I just tried to start it back up. Here's a link to

Re: [ceph-users] mirror OSD configuration

A more common search term for this might be Rack failure domain. The premise is the same for room as it is for rack, both can hold hosts and be set as the failure domain. There is a fair bit of discussion on how to achieve multi-rack/room/datacenter setups. Datacenter setups are more likely to

Re: [ceph-users] Cannot Create MGR

Could it be a problem that I have changed the hostname after the mon creation? What I mean is that # hostname -s ovhctrl # ceph daemon mon.$(hostname -s) quorum_status admin_socket: exception getting command descriptions: [Errno 2] No such file or directory But if I do it as

Re: [ceph-users] Ceph SNMP hooks?

You could probably write an SNMP module for the new ceph-mgr daemon. What do you want to use to monitor Ceph that requires SNMP? On Wed, Feb 28, 2018 at 1:13 PM Andre Goree wrote: > I've looked and haven't found much information besides custom 3rd-party > plugins so I figured

Re: [ceph-users] Cannot Create MGR

Hi, looks like you haven’t run the ceph-deploy command with the same user name and may be not the same current working directory. This could explain your problem. Make sure the other daemons have a mgr cap authorisation. You can find on this ML details about MGR caps being incorrect for OSDs

Re: [ceph-users] RBD mirroring to DR site

On Wed, Feb 28, 2018 at 2:56 PM, Brady Deetz wrote: > Great. We are read heavy. I assume the journals do not replicate reads. Is > that correct? Correct -- only writes (plus discards, snapshots, etc) are replicated. > On Wed, Feb 28, 2018 at 1:50 PM, Jason Dillaman

Re: [ceph-users] RBD mirroring to DR site

Great. We are read heavy. I assume the journals do not replicate reads. Is that correct? On Wed, Feb 28, 2018 at 1:50 PM, Jason Dillaman wrote: > On Wed, Feb 28, 2018 at 2:42 PM, Brady Deetz wrote: > > I'm considering doing one-way rbd mirroring to a DR

Re: [ceph-users] RBD mirroring to DR site

On Wed, Feb 28, 2018 at 2:42 PM, Brady Deetz wrote: > I'm considering doing one-way rbd mirroring to a DR site. The documentation > states that my link to the DR site should have sufficient throughput to > support replication. > > Our write activity is bursty. As such, we tend

Re: [ceph-users] Cannot Create MGR

I am still trying to figure what is the problem here... Initially the cluster was updated ok... # ceph health detail HEALTH_WARN noout flag(s) set; all OSDs are running luminous or later but require_osd_release < luminous; no active mgr noout flag(s) set all OSDs are running luminous or

[ceph-users] RBD mirroring to DR site

I'm considering doing one-way rbd mirroring to a DR site. The documentation states that my link to the DR site should have sufficient throughput to support replication. Our write activity is bursty. As such, we tend to see moments of high throughput 4-6gbps followed by long bouts of basically no

Re: [ceph-users] Ceph iSCSI is a prank?

Does Xen still not support RBD? Ceph has been around for years now! Met vriendelijke groeten, --  Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK: 61527076 | http://www.tuxis.nl/ T: 0318 200208 | i...@tuxis.nl Van: Massimiliano

Re: [ceph-users] Cannot Create MGR

OK...now this is getting crazy... data: pools: 0 pools, 0 pgs objects: 0 objects, 0 bytes usage: 0 kB used, 0 kB / 0 kB avail pgs: Where has gone everything?? What's happening here? G. Indeed John, you are right! I have updated "ceph-deploy" (which was installed

[ceph-users] Ceph SNMP hooks?

I've looked and haven't found much information besides custom 3rd-party plugins so I figured I'd ask here: Is there a way to monitor a clusters 'health' via SNMP? -- Andre Goree -=-=-=-=-=- Email - andre at drenet.net Website - http://blog.drenet.net PGP key -

Re: [ceph-users] OSD maintenance (ceph osd set noout)

On 2018/02/27 4:23 pm, John Spray wrote: On Tue, Feb 27, 2018 at 6:37 PM, Andre Goree wrote: Is it still considered best practice to set 'noout' for OSDs that will be going under maintenance, e.g., rebooting an OSD ndoe for a kernel update? I ask, because I've set this

Re: [ceph-users] Cannot Create MGR

Indeed John, you are right! I have updated "ceph-deploy" (which was installed via "pip" that's why wasn't updated with the rest ceph packages) but now it complaints that keys are missing $ ceph-deploy mgr create controller [ceph_deploy.conf][DEBUG ] found configuration file at:

Re: [ceph-users] Cannot Create MGR

On Wed, Feb 28, 2018 at 5:21 PM, Georgios Dimitrakakis wrote: > All, > > I have updated my test ceph cluster from Jewer (10.2.10) to Luminous > (12.2.4) using CentOS packages. > > I have updated all packages, restarted all services with the proper order > but I get a warning

Re: [ceph-users] mirror OSD configuration

On Wed, Feb 28, 2018 at 3:02 AM Zoran Bošnjak < zoran.bosn...@sloveniacontrol.si> wrote: > I am aware of monitor consensus requirement. It is taken care of (there is > a third room with only monitor node). My problem is about OSD redundancy, > since I can only use 2 server rooms for OSDs. > > I

[ceph-users] Cannot Create MGR

All, I have updated my test ceph cluster from Jewer (10.2.10) to Luminous (12.2.4) using CentOS packages. I have updated all packages, restarted all services with the proper order but I get a warning that the Manager Daemon doesn't exist. Here is the output: # ceph -s cluster: id:

Re: [ceph-users] Sizing your MON storage with a large cluster

Hi Wido, Are your mon's using rocksdb or still leveldb? Are your mon stores trimming back to a small size after HEALTH_OK was restored? One v12.2.2 cluster here just started showing the "is using a lot of disk space" warning on one of our mons. In fact all three mons are now using >16GB. I

Re: [ceph-users] Ceph iSCSI is a prank?

My thought is that in 4 years you could have migrated to a hypervisor that will have better performance into ceph than an added iSCSI layer. I won't deploy VMs for ceph on anything that won't allow librbd to work. Anything else is added complexity and reduced performance. On Wed, Feb 28, 2018,

Re: [ceph-users] Ceph-Fuse and mount namespaces

Am 28.02.2018 um 16:09 schrieb Patrick Donnelly: > On Tue, Feb 27, 2018 at 3:27 PM, Oliver Freyermuth > wrote: >> As you can see: >> - Name collision for admin socket, since the helper is already running. > > You can change the admin socket path using the `admin

Re: [ceph-users] Ceph iSCSI is a prank?

On Wed, Feb 28, 2018 at 9:17 AM, Max Cuttins wrote: > Sorry for being rude Ross, > > I follow Ceph since 2014 waiting for iSCSI support in order to use it with > Xen. What OS are you using in Dom0 that you cannot just directly use krbd? iSCSI is going to add an extra hop so

[ceph-users] Slow clients after git pull

Hello, I've created a Ceph cluster with 3 nodes and a FS to serve a webpage. The webpage speed is good enough (near to NFS speed), and have HA if one FS die. My problem comes when I deploy a git repository on that FS. The server makes a lot of IOPS to check the files that have to update and then

Re: [ceph-users] Ceph iSCSI is a prank?

Max, I understand your frustration. However, last time I checked, ceph was open source. Some of you might not remember, but one major reason why open source is great is that YOU CAN DO your own modifications. If you need a change like iSCSI support and it isn't there, it is probably best, if

[ceph-users] Luminous v12.2.4 released

This is the fourth bugfix release of Luminous v12.2.x long term stable release series. This was primarily intended to fix a few build, ceph-volume/ceph-disk issues from 12.2.3 and a few RGW issues. We recommend all the users of 12.2.x series to update. A full changelog is also published at the

Re: [ceph-users] Ceph iSCSI is a prank?

On Wed, Feb 28, 2018 at 10:06 AM, Max Cuttins wrote: > > > Il 28/02/2018 15:19, Jason Dillaman ha scritto: >> >> On Wed, Feb 28, 2018 at 7:53 AM, Massimiliano Cuttini >> wrote: >>> >>> I was building ceph in order to use with iSCSI. >>> But I just see from

Re: [ceph-users] reweight-by-utilization reverse weight after adding new nodes?

Thank you for your recommendation. On Mon, Feb 26, 2018 at 5:03 PM, David Turner wrote: > I would recommend continuing from where you are now and running `ceph osd > reweight-by-utilization` again. Your weights might be a little more odd, > but your data distribution

Re: [ceph-users] Ceph iSCSI is a prank?

On Feb 28, 2018 10:06 AM, "Max Cuttins" wrote: Il 28/02/2018 15:19, Jason Dillaman ha scritto: > On Wed, Feb 28, 2018 at 7:53 AM, Massimiliano Cuttini > wrote: > >> I was building ceph in order to use with iSCSI. >> But I just see from the docs that

Re: [ceph-users] Memory leak in Ceph OSD?

Hi Stefan, can you disable compression and check if memory is still leaking. If it stops then the issue is definitely somewhere along the "compress" path. Thanks, Igor On 2/28/2018 6:18 PM, Stefan Kooman wrote: Hi, TL;DR: we see "used" memory grows indefinitely on our OSD servers.

[ceph-users] Memory leak in Ceph OSD?

Hi, TL;DR: we see "used" memory grows indefinitely on our OSD servers. Until the point that either 1) a OSD process gets killed by OOMkiller, or 2) OSD aborts (proably because malloc cannot provide more RAM). I suspect a memory leak of the OSDs. We were running 12.2.2. We are now running 12.2.3.

Re: [ceph-users] Ceph-Fuse and mount namespaces

On Tue, Feb 27, 2018 at 3:27 PM, Oliver Freyermuth wrote: > As you can see: > - Name collision for admin socket, since the helper is already running. You can change the admin socket path using the `admin socket` config variable. Use metavariables [1] to make the

Re: [ceph-users] Ceph iSCSI is a prank?

Il 28/02/2018 15:19, Jason Dillaman ha scritto: On Wed, Feb 28, 2018 at 7:53 AM, Massimiliano Cuttini wrote: I was building ceph in order to use with iSCSI. But I just see from the docs that need: CentOS 7.5 (which is not available yet, it's still at 7.4)

Re: [ceph-users] Increase recovery / backfilling speed (with many small objects)

Hi, We recently learned on this list about the "rotational_journal = 1" for some (all?) NVMe / SSD setups. We also hit this issue (see below). It would eventually take a week to recover ... This was all "scratch data" so didn't matter anyway. We recently had to do some reovery / backfilling on

[ceph-users] Kernel problem with NBD resize

The rbd-nbd device no longer resizes online due to this bug: https://lkml.org/lkml/2018/2/18/85 My tracker below, but this is not a Ceph issue. https://tracker.ceph.com/issues/23137#change-108183 Has anyone heard any news and has any ability to inquire of the status of the fix? I don't see

Re: [ceph-users] Ceph-Fuse and mount namespaces

Am 28.02.2018 um 15:18 schrieb David Turner: > If you run your container in privileged mode you can mount ceph-fuse inside > of the VMs instead of from the shared resource on the host. I used a > configuration like this to test multi-tenency speed tests of CephFS using > ceph-fuse. The more

Re: [ceph-users] Ceph iSCSI is a prank?

On Wed, Feb 28, 2018 at 7:53 AM, Massimiliano Cuttini wrote: > I was building ceph in order to use with iSCSI. > But I just see from the docs that need: > > CentOS 7.5 > (which is not available yet, it's still at 7.4) > https://wiki.centos.org/Download > > Kernel 4.17 >

Re: [ceph-users] Ceph-Fuse and mount namespaces

If you run your container in privileged mode you can mount ceph-fuse inside of the VMs instead of from the shared resource on the host. I used a configuration like this to test multi-tenency speed tests of CephFS using ceph-fuse. The more mount points I used 1 per container, the more bandwidth I

Re: [ceph-users] Ceph iSCSI is a prank?

Sorry for being rude Ross, I follow Ceph since 2014 waiting for iSCSI support in order to use it with Xen. When finally it seemds it was implemented the OS requirements are irrealistic. Seems a bad prank. 4 year waiting for this... and still not true support yet. Il 28/02/2018 14:11,

Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

On 27 Feb 2018 06:46, "Jan Pekař - Imatic" wrote: I think I hit the same issue. I have corrupted data on cephfs and I don't remember the same issue before Luminous (i did the same tests before). It is on my test 1 node cluster with lower memory then recommended (so server

Re: [ceph-users] OSD maintenance (ceph osd set noout)

Like John says, noout prevents an osd being marked out in the cluster. It does not impede it from being marked down and back up which is the desired behavior when restarting a server. What are you seeing with your osds becoming unusable and needing to rebuild them? When rebooting a server if it

Re: [ceph-users] ceph mgr balancer bad distribution

Am 28.02.2018 um 13:58 schrieb Dan van der Ster: > Hi Stefan, > > Which balancer mode are you using? crush-compat scores using a mix of > nobjects, npgs, and size. It's doing pretty well over here as long as > you have a relatively small number of empty PGs. > > I believe that upmap uses nPGs

Re: [ceph-users] ceph mgr balancer bad distribution

Am 28.02.2018 um 13:59 schrieb John Spray: > On Wed, Feb 28, 2018 at 12:47 PM, Stefan Priebe - Profihost AG > wrote: >> Hello, >> >> with jewel we always used the python crush optimizer which gave us a >> pretty good distribution fo the used space. >> >> Since luminous

Re: [ceph-users] Random health OSD_SCRUB_ERRORS on various OSDs, after pg repair back to HEALTH_OK

Hi I read the bugtracker issue and it seems a lot like my problem, even if I can't check the reported checksum because I don't have it in my logs, perhaps it's because of debug osd = 0/0 in ceph.conf I just raised the OSD log level ceph tell osd.* injectargs --debug-osd 5/5 I'll check OSD

Re: [ceph-users] Ceph iSCSI is a prank?

Hi Massimiliano, have an espresso. You know the indians have a nice saying "Everything will be good at the end. If it is not good, it is still not the end." -Original Message- From: Massimiliano Cuttini [mailto:m...@phoenixweb.it] Sent: woensdag 28 februari 2018 13:53 To:

Re: [ceph-users] ceph mgr balancer bad distribution

On Wed, Feb 28, 2018 at 12:47 PM, Stefan Priebe - Profihost AG wrote: > Hello, > > with jewel we always used the python crush optimizer which gave us a > pretty good distribution fo the used space. > > Since luminous we're using the included ceph mgr balancer but the >

Re: [ceph-users] ceph mgr balancer bad distribution

Hi Stefan, Which balancer mode are you using? crush-compat scores using a mix of nobjects, npgs, and size. It's doing pretty well over here as long as you have a relatively small number of empty PGs. I believe that upmap uses nPGs only, and I haven't tested it enough yet to know if it actually

[ceph-users] Ceph iSCSI is a prank?

I was building ceph in order to use with iSCSI. But I just see from the docs that need: *CentOS 7.5* (which is not available yet, it's still at 7.4) https://wiki.centos.org/Download *Kernel 4.17* (which is not available yet, it is still at 4.15.7) https://www.kernel.org/ So I

[ceph-users] ceph mgr balancer bad distribution

Hello, with jewel we always used the python crush optimizer which gave us a pretty good distribution fo the used space. Since luminous we're using the included ceph mgr balancer but the distribution is far from perfect and much worse than the old method. Is there any way to tune the mgr

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

On Wed, Feb 28, 2018 at 11:05 AM, John Spray wrote: > On Wed, Feb 28, 2018 at 9:37 AM, Dan van der Ster wrote: >> Hi all, >> >> I'm just updating our test cluster from 12.2.2 to 12.2.4. Mon's and >> OSD's updated fine. >> >> When updating the MDS's (we

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

On Wed, Feb 28, 2018 at 9:37 AM, Dan van der Ster wrote: > Hi all, > > I'm just updating our test cluster from 12.2.2 to 12.2.4. Mon's and > OSD's updated fine. > > When updating the MDS's (we have 2 active and 1 standby), I started > with the standby. > > At the moment the

Re: [ceph-users] mirror OSD configuration

I am aware of monitor consensus requirement. It is taken care of (there is a third room with only monitor node). My problem is about OSD redundancy, since I can only use 2 server rooms for OSDs. I could use EC-pools, lrc or any other ceph configuration. But I could not find a configuration

Re: [ceph-users] Random health OSD_SCRUB_ERRORS on various OSDs, after pg repair back to HEALTH_OK

Hi, might be http://tracker.ceph.com/issues/22464 Can you check the OSD log file to see if the reported checksum is 0x6706be76? Paul > Am 28.02.2018 um 11:43 schrieb Marco Baldini - H.S. Amiata > : > > Hello > > I have a little ceph cluster with 3 nodes, each with

[ceph-users] Random health OSD_SCRUB_ERRORS on various OSDs, after pg repair back to HEALTH_OK

Hello I have a little ceph cluster with 3 nodes, each with 3x1TB HDD and 1x240GB SSD. I created this cluster after Luminous release, so all OSDs are Bluestore. In my crush map I have two rules, one targeting the SSDs and one targeting the HDDs. I have 4 pools, one using the SSD rule and the

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

On Wed, Feb 28, 2018 at 11:38 AM, Patrick Donnelly wrote: > On Wed, Feb 28, 2018 at 2:07 AM, Dan van der Ster wrote: >> (Sorry to spam) >> >> I guess it's related to this fix to the layout v2 feature id: >> https://github.com/ceph/ceph/pull/18782/files

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

On Wed, Feb 28, 2018 at 2:07 AM, Dan van der Ster wrote: > (Sorry to spam) > > I guess it's related to this fix to the layout v2 feature id: > https://github.com/ceph/ceph/pull/18782/files > > -#define MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 CompatSet::Feature(8, > "file layout

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

(Sorry to spam) I guess it's related to this fix to the layout v2 feature id: https://github.com/ceph/ceph/pull/18782/files -#define MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 CompatSet::Feature(8, "file layout v2") +#define MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 CompatSet::Feature(9, "file layout v2")

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

More: here is the MDS_FEATURES map for a running 12.2.2 cluster: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} and here it

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

Quoting Dan van der Ster (d...@vanderster.com): > Hi all, > > I'm just updating our test cluster from 12.2.2 to 12.2.4. Mon's and > OSD's updated fine. 12.2.4? Did you mean 12.2.3? Or did I miss something? Gr. stefan -- | BIT BV http://www.bit.nl/Kamer van Koophandel 09090351 | GPG:

[ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

Hi all, I'm just updating our test cluster from 12.2.2 to 12.2.4. Mon's and OSD's updated fine. When updating the MDS's (we have 2 active and 1 standby), I started with the standby. At the moment the standby MDS restarted into 12.2.4 [1], both active MDSs (still running 12.2.2) suicided like

Re: [ceph-users] ceph-deploy won't install luminous (but Jewel instead)

This worked. However somebody should investigate why default is still jewel on Centos 7.4 Il 28/02/2018 00:53, jorpilo ha scritto: Try using: ceph-deploy --release luminous host1... Mensaje original De: Massimiliano Cuttini Fecha: 28/2/18 12:42 a. m.