Re: [ceph-users] New OSD with weight 0, rebalance still happen...

On 11/22/18 6:12 PM, Marco Gaiarin wrote: Mandi! Paweł Sadowsk In chel di` si favelave... From your osd tree it looks like you used 'ceph osd reweight'. Yes, and i supposed also to do the right things! Now, i've tried to lower the to-dimissi OSD, using: ceph osd reweight 2 0.95

Re: [ceph-users] RBD image "lightweight snapshots"

On 08/10/2018 06:24 PM, Gregory Farnum wrote: On Fri, Aug 10, 2018 at 4:53 AM, Paweł Sadowsk wrote: On 08/09/2018 04:39 PM, Alex Elder wrote: On 08/09/2018 08:15 AM, Sage Weil wrote: On Thu, 9 Aug 2018, Piotr Dałek wrote: Hello, At OVH we're heavily utilizing snapshots for our backup

Re: [ceph-users] High TCP retransmission rates, only with Ceph

On 04/15/2018 08:18 PM, Robert Stanford wrote:  Iperf gives about 7Gb/s between a radosgw host and one of my OSD hosts (8 disks, 8 OSD daemons, one of 3 OSD hosts).  When I benchmark radosgw with cosbench I see high TCP retransmission rates (from sar -n ETCP 1).  I don't see this with iperf. 

Re: [ceph-users] Flash for mon nodes ?

On 06/21/2017 12:38 PM, Osama Hasebou wrote: > Hi Guys, > > Has anyone used flash SSD drives for nodes hosting Monitor nodes only? > > If yes, any major benefits against just using SAS drives ? We are using such setup for big (>500 OSDs) clusters. It makes it less painful when such cluster

Re: [ceph-users] Sparse file info in filestore not propagated to other OSDs

On 04/13/2017 04:23 PM, Piotr Dałek wrote: > On 04/06/2017 03:25 PM, Sage Weil wrote: >> On Thu, 6 Apr 2017, Piotr Dałek wrote: >>> Hello, >>> >>> We recently had an interesting issue with RBD images and filestore >>> on Jewel >>> 10.2.5: >>> We have a pool with RBD images, all of them mostly

Re: [ceph-users] SIGHUP to ceph processes every morning

Hi, 6:25 points to daily cron job, it's probably logrotate trying to force ceph to reopen logs. On 01/26/2017 07:34 AM, Torsten Casselt wrote: > Hi, > > I get the following line in journalctl: > > Jan 24 06:25:02 ceph01 ceph-osd[28398]: 2017-01-24 06:25:02.302770 > 7f0655516700 -1 received

Re: [ceph-users] After OSD Flap - FAILED assert(oi.version == i->first)

Hi, We see this error on Hammer 0.94.6. Bug report updated with logs. Thanks, On 11/15/2016 07:30 PM, Samuel Just wrote: > http://tracker.ceph.com/issues/17916 > > I just pushed a branch wip-17916-jewel based on v10.2.3 with some > additional debugging. Once it builds, would you be able to

Re: [ceph-users] effectively reducing scrub io impact

You can inspect source code or do: ceph --admin-daemon /var/run/ceph/ceph-osd.OSD_ID.asok config show | grep scrub # or similar And then check in source code :) On 10/20/2016 03:03 PM, Oliver Dzombic wrote: > Hi Christian, > > thank you for your time. > > The problem is deep scrub only. > >

Re: [ceph-users] unfound objects blocking cluster, need help!

Hi, I work with Tomasz and I'm investigating this situation. We still don't fully understood why there was unfound object after removing single OSD. >From logs[1] it looks like all PGs were active+clean before marking that OSD out. After that backfills started on multiple OSDs. Three minutes

Re: [ceph-users] Inconsistent PGs

to recover that pg. > > 2016-06-21 19:09 GMT+08:00 Paweł Sadowski <c...@sadziu.pl > <mailto:c...@sadziu.pl>>: > > Already restarted those OSD and then whole cluster (rack by rack, > failure domain is rack in this setup). > We would like to try *

Re: [ceph-users] Inconsistent PGs

uck stale > ceph pg dump_stuck inactive > ceph pg dump_stuck unclean > === > > And the query the PG, which are in unclean or stale state, check for > any issue with a specific OSD. > > Thanks > Swami > > On Tue, Jun 21, 2016 at 3:02 PM, Paweł Sadowski <c...@sadziu

Re: [ceph-users] Inconsistent PGs

? On 06/21/2016 12:37 PM, M Ranga Swami Reddy wrote: > Try to restart OSD 109 and 166? check if it help? > > > On Tue, Jun 21, 2016 at 4:05 PM, Paweł Sadowski <c...@sadziu.pl> wrote: >> Thanks for response. >> >> All OSDs seems to be ok, they have

[ceph-users] Inconsistent PGs

Hello, We have an issue on one of our clusters. One node with 9 OSD was down for more than 12 hours. During that time cluster recovered without problems. When host back to the cluster we got two PGs in incomplete state. We decided to mark OSDs on this host as out but the two PGs are still in

[ceph-users] RadosGW and X-Storage-Url

Hi, I'm testing RadosGW on Infernalis (9.2.1) and have two questions regarding X-Storage-Url header. First thing is that it always returns something like below: X-Storage-Url: http://my.example.domain:0/swift/v1 While docs say it should return "... {api version}/{account} prefix" Second

Re: [ceph-users] Intel P3700 PCI-e as journal drives?

Hi, Quick results for 1/5/10 jobs: # fio --filename=/dev/nvme0n1 --direct=1 --sync=1 --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test journal-test: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1 fio-2.1.3 Starting 1

Re: [ceph-users] O_DIRECT on deep-scrub read

On 10/07/2015 10:52 PM, Sage Weil wrote: > On Wed, 7 Oct 2015, David Zafman wrote: >> There would be a benefit to doing fadvise POSIX_FADV_DONTNEED after >> deep-scrub reads for objects not recently accessed by clients. > Yeah, it's the 'except for stuff already in cache' part that we don't do >

[ceph-users] O_DIRECT on deep-scrub read

Hi, Can anyone tell if deep scrub is done using O_DIRECT flag or not? I'm not able to verify that in source code. If not would it be possible to add such feature (maybe config option) to help keeping Linux page cache in better shape? Thanks, -- PS

Re: [ceph-users] Lot of blocked operations

On 09/18/2015 12:17 PM, Olivier Bonvalet wrote: > Le vendredi 18 septembre 2015 à 12:04 +0200, Jan Schermer a écrit : >>> On 18 Sep 2015, at 11:28, Christian Balzer wrote: >>> >>> On Fri, 18 Sep 2015 11:07:49 +0200 Olivier Bonvalet wrote: >>> Le vendredi 18 septembre 2015 à

Re: [ceph-users] Hammer reduce recovery impact

On 09/10/2015 10:56 PM, Robert LeBlanc wrote: > Things I've tried: > > * Lowered nr_requests on the spindles from 1000 to 100. This reduced > the max latency sometimes up to 3000 ms down to a max of 500-700 ms. > it has also reduced the huge swings in latency, but has also reduced > throughput

Re: [ceph-users] Erasure coded pools and bit-rot protection

://tracker.ceph.com/issues/12000 On Fri, Jun 12, 2015 at 12:32 PM, Gregory Farnum g...@gregs42.com wrote: On Fri, Jun 12, 2015 at 1:07 AM, Paweł Sadowski c...@sadziu.pl wrote: Hi All, I'm testing erasure coded pools. Is there any protection from bit-rot errors on object read? If I modify one bit

[ceph-users] Erasure coded pools and bit-rot protection

Hi All, I'm testing erasure coded pools. Is there any protection from bit-rot errors on object read? If I modify one bit in object part (directly on OSD) I'm getting *broken*object: mon-01:~ # rados --pool ecpool get `hostname -f`_16 - | md5sum bb2d82bbb95be6b9a039d135cc7a5d0d - #

[ceph-users] osd_scrub_sleep, osd_scrub_chunk_{min,max}

Hello Everyone, There are some options[1] that greatly reduces deep-scrub performance impact but they are not documented anywhere. Is there any reason for this? 1: - osd_scrub_sleep - osd_scrub_chunk_min - osd_scrub_chunk_max -- PS ___

[ceph-users] osd id == 2147483647 (2^31 - 1)

Has anyone saw something like this: osd id == 2147483647 (2147483647 == 2^31 - 1). Looks like some int casting bug but I have no idea where to look for it (and I don't know exact steps to reproduce this - I was just doing osd in/osd out multiple times to test recovery speed under some client

Re: [ceph-users] osd id == 2147483647 (2^31 - 1)

as 2147483647 and that's what you have in [7,2,2147483647,6,10] in the case of a replicated pool, the missing OSDs would be omitted instead. In Hammer 2147483647 shows as NONE which is less confusing. Cheers On 26/05/2015 09:16, Paweł Sadowski wrote: Has anyone saw something like this: osd id

Re: [ceph-users] Ceph data consistency

On 12/30/2014 09:40 AM, Chen, Xiaoxi wrote: Hi, First of all, the data is safe since it's persistent in journal, if error occurs on OSD data partition, replay the journal will get the data back. Agree. Data are safe in journal. But when journal is flushed data are moved to a filestore and

Re: [ceph-users] Ceph inconsistency after deep-scrub

On 11/21/2014 10:46 PM, Paweł Sadowski wrote: W dniu 21.11.2014 o 20:12, Gregory Farnum pisze: On Fri, Nov 21, 2014 at 2:35 AM, Paweł Sadowski c...@sadziu.pl wrote: Hi, During deep-scrub Ceph discovered some inconsistency between OSDs on my cluster (size 3, min size 2). I have fund broken

[ceph-users] Ceph inconsistency after deep-scrub

Hi, During deep-scrub Ceph discovered some inconsistency between OSDs on my cluster (size 3, min size 2). I have fund broken object and calculated md5sum of it on each OSD (osd.195 is acting_primary): osd.195 - md5sum_ osd.40 - md5sum_ osd.314 - md5sum_ I run ceph pg repair and

[ceph-users] osd_disk_thread_ioprio_class/_priorioty ignored?

Hi, I was trying to determine performance impact of deep-scrubbing with osd_disk_thread_ioprio_class option set but it looks like it's ignored. Performance (during deep-scrub) is the same with this options set or left with defaults (1/3 of normal performance). # ceph --admin-daemon

Re: [ceph-users] osd_disk_thread_ioprio_class/_priorioty ignored?

On 10/23/2014 09:10 AM, Paweł Sadowski wrote: Hi, I was trying to determine performance impact of deep-scrubbing with osd_disk_thread_ioprio_class option set but it looks like it's ignored. Performance (during deep-scrub) is the same with this options set or left with defaults (1/3 of normal

[ceph-users] Weight of new OSD

Hi, From time to time when I replace broken OSD the new one gets weight of zero. Crush map from epoch before adding OSD seems to be fine. Is there any way to debug this issue? Regards, -- PS ___ ceph-users mailing list ceph-users@lists.ceph.com

[ceph-users] ceph-users@lists.ceph.com

Hi, I'm trying to start Qemu on top of RBD. In documentation[1] there is a big warning: Important If you set rbd_cache=true, you must set cache=writeback or risk data loss. Without cache=writeback, QEMU will not send flush requests to librbd. If QEMU exits uncleanly in this

Re: [ceph-users] Ceph + Qemu cache=writethrough

Sorry for missing subject. On 08/21/2014 03:09 PM, Paweł Sadowski wrote: Hi, I'm trying to start Qemu on top of RBD. In documentation[1] there is a big warning: Important If you set rbd_cache=true, you must set cache=writeback or risk data loss. Without cache=writeback, QEMU

Re: [ceph-users] Cache tiering and target_max_bytes

On 08/14/2014 10:30 PM, Sage Weil wrote: On Thu, 14 Aug 2014, Pawe? Sadowski wrote: W dniu 14.08.2014 17:20, Sage Weil pisze: On Thu, 14 Aug 2014, Pawe? Sadowski wrote: Hello, I've a cluster of 35 OSD (30 HDD, 5 SSD) with cache tiering configured. During tests it looks like ceph is not

[ceph-users] Cache tiering and target_max_bytes

Hello, I've a cluster of 35 OSD (30 HDD, 5 SSD) with cache tiering configured. During tests it looks like ceph is not respecting target_max_bytes settings. Steps to reproduce: - configure cache tiering - set target_max_bytes to 32G (on hot pool) - write more than 32G of data - nothing happens

Re: [ceph-users] Cache tiering and target_max_bytes

W dniu 14.08.2014 17:20, Sage Weil pisze: On Thu, 14 Aug 2014, Pawe? Sadowski wrote: Hello, I've a cluster of 35 OSD (30 HDD, 5 SSD) with cache tiering configured. During tests it looks like ceph is not respecting target_max_bytes settings. Steps to reproduce: - configure cache tiering -