[ceph-users] RGW pubsub deprecation

2020-11-04 Thread Yuval Lifshitz
Dear Community, Since Nautilus, we have 2 mechanisms for notifying 3rd parties on changes in buckets and objects: "bucket notifications" [1] and "pubsub" [2]. In "bucket notifications" (="push mode") the events are sent from the RGW to an external entity (kafka, rabbitmq etc.), while in "pubsub" (

[ceph-users] Fwd: File read are not completing and IO shows in bytes able to not reading from cephfs

2020-11-04 Thread Amudhan P
Update on the issue of reading file drops to bytes and error. When new files are copied to mount it works fine and reading the same also working with no issue. But reading old or existing files still the same issue and below error msg in client. "libceph: osd1 10.0.104.1:6891 socket closed (con st

[ceph-users] Re: Ceph flash deployment

2020-11-04 Thread Frank Schilder
That's actually an interesting question. On the 5.9 kernels cfq seems not available: # cat /sys/block/sdj/queue/scheduler [mq-deadline] kyber bfq none What is the recommendation here? Best regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 __

[ceph-users] RBD image stuck and no erros on logs

2020-11-04 Thread Salsa
Hi, This same error keeps happening to me: after writing some amount of data to an RBD image it gets stuck and no read or write operation on it works. Every operation hangs. I cannot resize, alter features, read or write data. I can mount it, but using parted or fdisk hangs indefinitely. In the

[ceph-users] Re: Ceph flash deployment

2020-11-04 Thread Seena Fallah
I see in this thread that someone is saying that bluestore is only works good with cfq scheduler: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-November/031063.html For readahead, do you have any measurements to see how I can measure my workload to see if I should increase it or not? T

[ceph-users] Mon went down and won't come back

2020-11-04 Thread Paul Mezzanini
Hi everyone, I figure it's time to pull in more brain power on this one. We had an NVMe mostly die in one of our monitors and it caused the write latency for the machine to spike. Ceph did the RightThing(tm) and when it lost quorum on that machine it was ignored. I pulled the bad drive out o

[ceph-users] Re: Cephadm: module not found

2020-11-04 Thread Nadiia Kotelnikova
I have updated dockers to version "v15.2.5", but it still did not resolve the issue. Luckily, i found the solution, which is strange: i had in config "mgr/cephadm/log_to_file" set to true. After i removed this option, i am able to execute commands, related to orchestrator: > sudo ceph conf

[ceph-users] Re: Seriously degraded performance after update to Octopus

2020-11-04 Thread Martin Rasmus Lundquist Hansen
Thank you for the suggestion. It does indeed seem to explain why the OSD nodes are no longer using the Buffers for caching. Unfortunately, changing the value bluefs_buffered_io does not seem to make any difference in performance. I will keep looking for clues. __

[ceph-users] Re: How to reset Log Levels

2020-11-04 Thread Ml Ml
Is this still debug output or "normal"?: Nov 04 10:19:39 ceph01 bash[2648]: audit 2020-11-04T09:19:38.577088+ mon.ceph03 (mon.0) 7738 : audit [DBG] from='mgr.42824785 10.10.2.103:0/3293316818' entity='mgr.ceph03' cmd=[{"prefix": "mds metadata", "who": "cephfs.ceph04.hrcvab"}]: dispatch Nov 04

[ceph-users] File read are not completing and IO shows in bytes able to not reading from cephfs

2020-11-04 Thread Amudhan P
Hi, In my test ceph octopus cluster I was trying to simulate a failure case of when client mounted cephfs thru kernel client and doing read and write process, shutting down entire cluster with OSD flags like no down, no out, no backfiling and no recovery. Cluster is 4 node composed of 3 mons, 2

[ceph-users] bluefs_buffered_io

2020-11-04 Thread Marcel Kuiper
Hi list, I see a few changes in the (minor) version changelogs in the default for bluefs_buffered_io setting. Sometimes it is set to true in our version (14.2.11) it is set to false Can someone shed a light on this setting? I fail to find any documentation on it. ceph config help is not entirely

[ceph-users] Re: Ceph 14.2 - some PGs stuck peering.

2020-11-04 Thread Eugen Block
Hi, it's not really clear what happened, I would investigate the root cause first. Did some of the OSDs fail, if yes, why? To increase the recovery speed you can change these values live: osd_max_backfills osd_recovery_max_active Choose carefully and only increase slowly as it can easily imp