Re: [ceph-users] how to judge the results? - rados bench comparison

Thanks Paul for the judgement. Tue, 16 Apr 2019 10:13:03 +0200 Paul Emmerich ==> Lars Täuber : > Seems in line with what I'd expect for the hardware. > > Your hardware seems to be way overspecced, you'd be fine with half the > RAM, half the CPU and way cheaper disks. Do you mean all the

Re: [ceph-users] Glance client and RBD export checksum mismatch

Dear All, Thanks for the input. We have created another pool and copied all the objects from images pool to it. New pool : images_new Once its done, have ran the script against the new pool and expected checksum received. So issue is only happening from current glance integrated pool only,

Re: [ceph-users] Is it possible to run a standalone Bluestore instance?

Thanks for your suggestions. I tried to build libfio_ceph_objectstore.so, but it fails to load: ``` $ LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so fio: engine libfio_ceph_objectstore.so not loadable IO engine libfio_ceph_objectstore.so not found ``` I managed to print

Re: [ceph-users] Try to log the IP in the header X-Forwarded-For with radosgw behind haproxy

Hi Francois, Why is using an explicit unix socket problematic for you? For what it does, that decision has always seemed sensible. Matt On Tue, Apr 16, 2019 at 7:04 PM Francois Lafont wrote: > > Hi @all, > > On 4/9/19 12:43 PM, Francois Lafont wrote: > > > I have tried this config: > > > >

Re: [ceph-users] Try to log the IP in the header X-Forwarded-For with radosgw behind haproxy

Hi @all, On 4/9/19 12:43 PM, Francois Lafont wrote: I have tried this config: - rgw enable ops log  = true rgw ops log socket path = /tmp/opslog rgw log http headers    = http_x_forwarded_for - and I have logs in the socket /tmp/opslog like this: -

Re: [ceph-users] showing active config settings

$ ceph config set osd osd_recovery_max_active 4 $ ceph daemon osd.0 config diff|grep -A5 osd_recovery_max_active "osd_recovery_max_active": { "default": 3, "mon": 4, "override": 4, "final": 4 }, On Wed, Apr 17, 2019 at 5:29 AM

Re: [ceph-users] showing active config settings

I wish there was a way to query the running settings from one of the MGR hosts, and it doesn't help that ansible doesn't even copy the keyring to the OSD nodes so commands there wouldn't work anyway. I'm still puzzled why it doesn't show any change when I run this no matter what I set it to: #

Re: [ceph-users] Cannot quiet "pools have many more objects per pg than average" warning

On Tue, Apr 16, 2019 at 1:28 PM Paul Emmerich wrote: > > I think the warning is triggered by the mgr daemon and not the mon, > try setting it there > Thank you Paul. How do I set it in the mgr daemon? I tried: ceph tell mon.* injectargs '--mgr_pg_warn_max_object_skew 0' ceph tell mgr.*

Re: [ceph-users] Limits of mds bal fragment size max

On Fri, Apr 12, 2019 at 10:31 AM Benjeman Meekhof wrote: > > We have a user syncing data with some kind of rsync + hardlink based > system creating/removing large numbers of hard links. We've > encountered many of the issues with stray inode re-integration as > described in the thread and

Re: [ceph-users] Cannot quiet "pools have many more objects per pg than average" warning

I think the warning is triggered by the mgr daemon and not the mon, try setting it there Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Tue, Apr 16, 2019 at 8:18

[ceph-users] Cannot quiet "pools have many more objects per pg than average" warning

Hi, I am getting a health warning about many more objects for PG than average. Seems to be common with RadosGW, where pools other than data contain very small number of objects. ceph@ola-s3-stg:/etc/ceph$ ceph health detail HEALTH_WARN 1 pools have many more objects per pg than average

Re: [ceph-users] NFS-Ganesha CEPH_FSAL | potential locking issue

On Tue, Apr 16, 2019 at 10:36 AM David C wrote: > > Hi All > > I have a single export of my cephfs using the ceph_fsal [1]. A CentOS 7 > machine mounts a sub-directory of the export [2] and is using it for the home > directory of a user (e.g everything under ~ is on the server). > > This works

Re: [ceph-users] Limiting osd process memory use in nautilus.

I am using osd_memory_target ( http://docs.ceph.com/docs/nautilus/rados/configuration/bluestore-config-ref/#automatic-cache-sizing) for this purpose and it works flawless. Kept working after upgrading from mimic to nautilus. Jonathan Proulx schrieb am Di., 16. Apr. 2019, 18:06: > Hi All, > > I

Re: [ceph-users] Limiting osd process memory use in nautilus.

As of 13.2.3, you should use 'osd_memory_target' instead of 'bluestore_cache_size' -- Adam On Tue, Apr 16, 2019 at 10:28 AM Jonathan Proulx wrote: > > Hi All, > > I have a a few servers that are a bit undersized on RAM for number of > osds they run. > > When we swithced to bluestore about 1yr

[ceph-users] Limiting osd process memory use in nautilus.

Hi All, I have a a few servers that are a bit undersized on RAM for number of osds they run. When we swithced to bluestore about 1yr ago I'd "fixed" this (well kept them from OOMing) by setting bluestore_cache_size_ssd and bluestore_cache_size_hdd, this worked. after upgrading to Nautilus the

Re: [ceph-users] Multi-site replication speed

Hi Brian, On 4/16/19 1:57 AM, Brian Topping wrote: On Apr 15, 2019, at 5:18 PM, Brian Topping > wrote: If I am correct, how do I trigger the full sync? Apologies for the noise on this thread. I came to discover the `radosgw-admin [meta]data sync init`

[ceph-users] NFS-Ganesha CEPH_FSAL | potential locking issue

Hi All I have a single export of my cephfs using the ceph_fsal [1]. A CentOS 7 machine mounts a sub-directory of the export [2] and is using it for the home directory of a user (e.g everything under ~ is on the server). This works fine until I start a long sequential write into the home

Re: [ceph-users] HW failure cause client IO drops

Hello, Are you using a BBU backed raid controller? It sounds more like your write cache is acting up if you are using one. Can you check what your raid controller is showing? I have sometimes seen raid controllers performing consistency checks or patrol read on single drive raid0. You can disable

Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic

On 4/15/2019 4:17 PM, Wido den Hollander wrote: On 4/15/19 2:55 PM, Igor Fedotov wrote: Hi Wido, the main driver for this backport were multiple complains on write ops latency increasing over time. E.g. see thread named:  "ceph osd commit latency increase over time, until restart" here. Or

Re: [ceph-users] Is it possible to run a standalone Bluestore instance?

Besides already mentioned store_test.cc one can also use ceph objectstore fio plugin (https://github.com/ceph/ceph/tree/master/src/test/fio) to access standalone BlueStore instance from FIO benchmarking tool. Thanks, Igor On 4/16/2019 7:58 AM, Can ZHANG wrote: Hi, I'd like to run a

Re: [ceph-users] HW failure cause client IO drops

On 4/16/19 2:27 PM, M Ranga Swami Reddy wrote: > Its Smart Storage battery, which was disabled due to high ambient > temperature. > All OSD processes/daemon working as is...but those OSDs not responding > to other OSD due to high CPU utilization.. > Don't observe the clock skew issue. > As the

Re: [ceph-users] HW failure cause client IO drops

Its Smart Storage battery, which was disabled due to high ambient temperature. All OSD processes/daemon working as is...but those OSDs not responding to other OSD due to high CPU utilization.. Don't observe the clock skew issue. On Tue, Apr 16, 2019 at 12:49 PM Marco Gaiarin wrote: > Mandi! M

Re: [ceph-users] Does "ceph df" use "bogus" copies factor instead of (k, m) for erasure coded pool?

On Tue, 16 Apr 2019 at 17:05, Paul Emmerich wrote: > > No, the problem is that a storage system should never tell a client > that it has written data if it cannot guarantee that the data is still > there if one device fails. [...] Ah, now I got your point. Anyways, it should be users' choice

Re: [ceph-users] Does "ceph df" use "bogus" copies factor instead of (k, m) for erasure coded pool?

No, the problem is that a storage system should never tell a client that it has written data if it cannot guarantee that the data is still there if one device fails. Scenario: one OSD is down for whatever reason and another one fails. You've now lost all writes that happened while one OSD was

Re: [ceph-users] Does "ceph df" use "bogus" copies factor instead of (k, m) for erasure coded pool?

On Tue, 16 Apr 2019 at 16:52, Paul Emmerich wrote: > On Tue, Apr 16, 2019 at 11:50 AM Igor Podlesny wrote: > > On Tue, 16 Apr 2019 at 14:46, Paul Emmerich wrote: [...] > > Looked at it, didn't see any explanation of your point of view. If > > there're 2 active data instances > > (and 3rd is

Re: [ceph-users] Does "ceph df" use "bogus" copies factor instead of (k, m) for erasure coded pool?

On Tue, Apr 16, 2019 at 11:50 AM Igor Podlesny wrote: > > On Tue, 16 Apr 2019 at 14:46, Paul Emmerich wrote: > > Sorry, I just realized I didn't answer your original question. > [...] > > No problemo. -- I've figured out the answer to my own question earlier > anyways. > And actually gave a

Re: [ceph-users] Does "ceph df" use "bogus" copies factor instead of (k, m) for erasure coded pool?

On Tue, 16 Apr 2019 at 14:46, Paul Emmerich wrote: > Sorry, I just realized I didn't answer your original question. [...] No problemo. -- I've figured out the answer to my own question earlier anyways. And actually gave a hint today

Re: [ceph-users] Fwd: HW failure cause client IO drops

OSD processes/daemon running as is...So ceph not making those OSD down or out. But as battery failed, which leads temperature high, leads CPU utlization increased - leads OSD response time more, so that other OSDs failed to response on time.. causing the utter slow or no IO... On Tue, Apr 16,

Re: [ceph-users] showing active config settings

On Tue, Apr 16, 2019 at 6:03 PM Paul Emmerich wrote: > > This works, it just says that it *might* require a restart, but this > particular option takes effect without a restart. We've already looked at changing the wording once to make it more palatable. http://tracker.ceph.com/issues/18424 >

Re: [ceph-users] how to judge the results? - rados bench comparison

Seems in line with what I'd expect for the hardware. Your hardware seems to be way overspecced, you'd be fine with half the RAM, half the CPU and way cheaper disks. In fakt, a good SATA 4kn disk can be faster than a SAS 512e disk. I'd probably only use the 25G network for both networks instead

Re: [ceph-users] showing active config settings

This works, it just says that it *might* require a restart, but this particular option takes effect without a restart. Implementation detail: this message shows up if there's no internal function to be called when this option changes, so it can't be sure if the change is actually doing anything

Re: [ceph-users] Is it possible to run a standalone Bluestore instance?

Depending on how low-level you are willing to go, a good place to start would be the unit test for the various object store implementations: https://github.com/ceph/ceph/blob/master/src/test/objectstore/store_test.cc Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us

Re: [ceph-users] Does "ceph df" use "bogus" copies factor instead of (k, m) for erasure coded pool?

Sorry, I just realized I didn't answer your original question. ceph df does take erasure coding settings into account and shows the correct free space. However, it also takes the current data distribution into account, i.e., the amount of data you can write until the first OSD is full assuming

Re: [ceph-users] HW failure cause client IO drops

Mandi! M Ranga Swami Reddy In chel di` si favelave... > Hello - Recevenlt we had an issue with storage node's battery failure, which > cause ceph client IO dropped to '0' bytes. Means ceph cluster couldn't perform > IO operations on the cluster till the node takes out. This is not expected >

Re: [ceph-users] 'Missing' capacity

root@proxmox01:~# ceph osd df tree | sort -n -k8 | tail -1   1   ssd  0.87000  1.0  889GiB  721GiB  168GiB 81.14 1.50  82         osd.1       root@proxmox01:~# ceph osd df tree | grep -c osd 68 68*168=11424 That is closer, thanks. I thought that available was the same as the cluster

Re: [ceph-users] Fwd: HW failure cause client IO drops

Good morning, the OSDs are usually marked out after 10 minutes, that's when rebalancing starts. But the I/O should not drop during that time, this could be related to your pool configuration. If you have a replicated pool of size 3 and also set min_size to 3 the I/O would pause if a node