Re: [ceph-users] Slow OPS

Actually, the lag is between "sub_op_committed" and "commit_sent". Is there any pattern to these slow requests? Do they involve the same osd, or set of osds? On Thu, Mar 21, 2019 at 3:37 PM Brad Hubbard wrote: > > On Thu, Mar 21, 2019 at 3:20 PM Glen Baars > wrote: > > > > Thanks for that - we

Re: [ceph-users] Slow OPS

On Thu, Mar 21, 2019 at 3:20 PM Glen Baars wrote: > > Thanks for that - we seem to be experiencing the wait in this section of the > ops. > > { > "time": "2019-03-21 14:12:42.830191", > "event": "sub_op_committed" >

Re: [ceph-users] Slow OPS

Thanks for that - we seem to be experiencing the wait in this section of the ops. { "time": "2019-03-21 14:12:42.830191", "event": "sub_op_committed" }, { "time":

Re: [ceph-users] SSD Recovery Settings

Lots of good info there, thank you! I tend to get options fatigue when trying to pick out a new system. This should help narrow that focus greatly. -Brent From: Reed Dier Sent: Wednesday, March 20, 2019 12:48 PM To: Brent Kennedy Cc: ceph-users Subject: Re: [ceph-users] SSD

[ceph-users] CephFS performance improved in 13.2.5?

Hello, Yesterday I upgraded from 13.2.2 to 13.2.5 and so far I have only seen significant improvements in MDS operations. Needless to say I'm happy, but I didn't notice anything related in release notes. Am I missing something, possibly new configuration settings? Screenshots below:

Re: [ceph-users] Slow OPS

On Thu, Mar 21, 2019 at 12:11 AM Glen Baars wrote: > > Hello Ceph Users, > > > > Does anyone know what the flag point ‘Started’ is? Is that ceph osd daemon > waiting on the disk subsystem? This is set by "mark_started()" and is roughly set when the pg starts processing the op. Might want to

Re: [ceph-users] ceph-osd 14.2.0 won't start: Failed to pick public address on IPv6 only cluster

On 20/03/2019 19:53, Ricardo Dias wrote: Make sure you have the following option in ceph.conf: ms_bind_ipv4 = false That will prevent the OSD from trying to find an IPv4 address. Thank you! I've only ever used ms_bind_ipv6 = true on its own. Adding your line solved my problem. Simon

Re: [ceph-users] ceph-osd 14.2.0 won't start: Failed to pick public address on IPv6 only cluster

Hi, Make sure you have the following option in ceph.conf: ms_bind_ipv4 = false That will prevent the OSD from trying to find an IPv4 address. Cheers, Ricardo Dias > On 20 Mar 2019, at 19:41, Simon Ironside wrote: > > Hi Everyone, > > I'm upgrading an IPv6 only cluster from 13.2.5 Mimic to

Re: [ceph-users] v14.2.0 Nautilus released

with Debian buster frozen, If there are issues with ceph on debian that would best be fixed in debian, now is the last chance to get anything into buster before the next release. it is also important to get mimic and luminous packages built for Buster. Since you want to avoid a situation

[ceph-users] ceph-osd 14.2.0 won't start: Failed to pick public address on IPv6 only cluster

Hi Everyone, I'm upgrading an IPv6 only cluster from 13.2.5 Mimic to 14.2.0 Nautilus. The mon and mgr upgrades went fine, the first OSD node unfortunately fails to restart after upgdating the packages. The affected ceph-osd logs show the lines: Unable to find any IPv4 address in networks

Re: [ceph-users] Looking up buckets in multi-site radosgw configuration

On Tue, Mar 19, 2019 at 7:51 AM Casey Bodley wrote: > Yeah, correct on both points. The zonegroup redirects would be the only > way to guide clients between clusters. Awesome. Thank you for the clarification. ___ ceph-users mailing list

Re: [ceph-users] SSD Recovery Settings

Grafana is the web frontend for creating the graphs. InfluxDB holds the time series data that Grafana pulls from. To collect data, I am using collectd daemons

Re: [ceph-users] SSD Recovery Settings

Seems both of you are spot on. I injected the change and its now moving at .080 instead of .002. I did fix the label on the drives from HDD to SDD but I didn't restart the OSDs due to the recovery process. Seeing it fly now. I also restarted the stuck OSDs but I know they are where the data is,

[ceph-users] cephfs manila snapshots best practices

Hi all, We're currently upgrading our cephfs (managed by OpenStack Manila) clusters to Mimic, and want to start enabling snapshots of the file shares. There are different ways to approach this, and I hope someone can share their experiences with: 1. Do you give users the 's' flag in their cap,

Re: [ceph-users] SSD Recovery Settings

Not sure what your OSD config looks like, When I was moving from Filestore to Bluestore on my SSD OSD's (and NVMe FS journal to NVMe Bluestore block.db), I had an issue where the OSD was incorrectly being reported as rotational in some part of the chain. Once I overcame that, I had a huge boost

[ceph-users] Slow OPS

Hello Ceph Users, Does anyone know what the flag point 'Started' is? Is that ceph osd daemon waiting on the disk subsystem? Ceph 13.2.4 on centos 7.5 "description": "osd_op(client.1411875.0:422573570 5.18ds0 5:b1ed18e5:::rbd_data.6.cf7f46b8b4567.0046e41a:head [read

Re: [ceph-users] fio test rbd - single thread - qd1

> `cpupower idle-set -D 0` will help you a lot, yes. > > However it seems that not only the bluestore makes it slow. >= 50% of the > latency is introduced by the OSD itself. I'm just trying to understand > WHAT parts of it are doing so much work. For example in my current case > (with cpupower

Re: [ceph-users] CephFS: effects of using hard links

On Tue, Mar 19, 2019 at 9:43 AM Erwin Bogaard wrote: > > Hi, > > > > For a number of application we use, there is a lot of file duplication. This > wastes precious storage space, which I would like to avoid. > > When using a local disk, I can use a hard link to let all duplicate files > point

Re: [ceph-users] CephFS: effects of using hard links

On 3/20/19 11:54 AM, Gregory Farnum wrote: On Tue, Mar 19, 2019 at 2:13 PM Erwin Bogaard > wrote: Hi, For a number of application we use, there is a lot of file duplication. This wastes precious storage space, which I would like to avoid.

Re: [ceph-users] fio test rbd - single thread - qd1

On 3/20/19 3:12 AM, Vitaliy Filippov wrote: `cpupower idle-set -D 0` will help you a lot, yes. However it seems that not only the bluestore makes it slow. >= 50% of the latency is introduced by the OSD itself. I'm just trying to understand WHAT parts of it are doing so much work. For example

Re: [ceph-users] ceph-volume lvm batch OSD replacement

On Tue, Mar 19, 2019 at 12:25 PM Dan van der Ster wrote: > > On Tue, Mar 19, 2019 at 12:17 PM Alfredo Deza wrote: > > > > On Tue, Mar 19, 2019 at 7:00 AM Alfredo Deza wrote: > > > > > > On Tue, Mar 19, 2019 at 6:47 AM Dan van der Ster > > > wrote: > > > > > > > > Hi all, > > > > > > > > We've

Re: [ceph-users] fio test rbd - single thread - qd1

`cpupower idle-set -D 0` will help you a lot, yes. However it seems that not only the bluestore makes it slow. >= 50% of the latency is introduced by the OSD itself. I'm just trying to understand WHAT parts of it are doing so much work. For example in my current case (with cpupower

Re: [ceph-users] fio test rbd - single thread - qd1

On 19/03/2019 16:17, jes...@krogh.cc wrote: Hi All. I'm trying to get head and tails into where we can stretch our Ceph cluster into what applications. Parallism works excellent, but baseline throughput it - perhaps - not what I would expect it to be. Luminous cluster running bluestore -

Re: [ceph-users] CEPH ISCSI LIO multipath change delay

On 20/03/2019 07:43, li jerry wrote: Hi,ALL I’ve deployed mimic(13.2.5) cluster on 3 CentOS 7.6 servers, then configured iscsi-target and created a LUN, referring to http://docs.ceph.com/docs/mimic/rbd/iscsi-target-cli/. I have another server which is CentOS 7.4, configured and mounted

[ceph-users] New Bluestore Cluster Hardware Questions

Hey all, we've been running Ceph for a while and I'm in the process of providing hardware for new site. On current site we run filestore-ceph and for the new site I'm going ahead for a bluestore-ceph cluster. This is what I've come up for hardware specification for the server, which we will

Re: [ceph-users] v14.2.0 Nautilus released

On Tue, Mar 19, 2019 at 2:53 PM Benjamin Cherian wrote: > > Hi, > > I'm getting an error when trying to use the APT repo for Ubuntu bionic. Does > anyone else have this issue? Is the mirror sync actually still in progress? > Or was something setup incorrectly? > > E: Failed to fetch >

Re: [ceph-users] v14.2.0 Nautilus released

There aren't any Debian packages built for this release because we haven't updated the infrastructure to build (and test) Debian packages yet. On Tue, Mar 19, 2019 at 10:24 AM Sean Purdy wrote: > > Hi, > > > Will debian packages be released? I don't see them in the nautilus repo. I > thought