Re: [ceph-users] Limit cross-datacenter network traffic during EC recovery

[Konstantin Shalygin]

so we can prevent high network throughput between DC1 and DC2
during rebuild because an entire OSD server fails? If yes, how?



set max_recovery, max_backfill or '_sleeps'. On other hands - network QoS.




k

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Proper procedure to replace DB/WAL SSD

[Jens-U. Mozdzen]

Hi *,

sorry for bringing up that old topic again, but we just faced a  
corresponding situation and have successfully tested two migration  
scenarios.


Zitat von ceph-users-requ...@lists.ceph.com:

Date: Sat, 24 Feb 2018 06:10:16 +
From: David Turner 
To: Nico Schottelius 
Cc: Caspar Smit , ceph-users

Subject: Re: [ceph-users] Proper procedure to replace DB/WAL SSD
Message-ID:

Content-Type: text/plain; charset="utf-8"

Caspar, it looks like your idea should work. Worst case scenario seems like
the osd wouldn't start, you'd put the old SSD back in and go back to the
idea to weight them to 0, backfilling, then recreate the osds. Definitely
with a try in my opinion, and I'd love to hear your experience after.

Nico, it is not possible to change the WAL or DB size, location, etc after
osd creation.


it is possible to move a separate WAL/DB to a new device, whilst  
without changing the size. We have done this for multiple OSDs, using  
only existing (mainstream :) ) tools and have documented the procedure  
in  
http://heiterbiswolkig.blogs.nde.ag/2018/04/08/migrating-bluestores-block-db/  
. It will *not* allow to separate WAL / DB after OSD creation, nor  
does it allow changing the DB size.


As we faced a failing WAL/DB SSD during one of the moves (fatal read  
errors from the DB block device), we also established a procedure to  
initialize the OSD to "empty" during that operation, so that the OSD  
will get re-filled without changing the OSD map:  
http://heiterbiswolkig.blogs.nde.ag/2018/04/08/resetting-an-existing-bluestore-osd/


HTH
Jens

PS: Live WAL/DB migration is something that can be done easily when  
using logical volumes, which is why I'd highly recommend to go that  
route, instead of using partitions. LVM not only helps when the SSDs  
reach their EOL, but with live changes to load balancing (WAL/DB LVs  
distributing across multiple SSDs), too.



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Limit cross-datacenter network traffic during EC recovery

[Systeembeheerder Nederland]

Hi,

Intended setup (only focusing on OSD server here, of course monitor and
other servers will be part of the intended cluster):
* 1 single Cluster, spanning across 2 datacenters:
* 6 OSD servers (each containing 5 OSD’s/disks) in Datacenter1
* 6 OSD servers (each containing 5 OSD’s/disks) in Datacenter2

Requirements:
* Erasure Code level k=3, m=2 (total overhead for the entire cluster should
be: 40%)
* ISA-L Erasure Code optimization
* Our requirements regarding IOPS, latency and throughput are low
* Survive failure of 2 OSD disks
* Survive failure of 1 OSD server

Question:
Is there a way to keep Erasuere Codes for the 6 OSD servers located in DC1
at DC1, so we can prevent high network throughput between DC1 and DC2
during rebuild because an entire OSD server fails? If yes, how?

Best regards,
Karel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[ceph]

Am 8. April 2018 05:44:11 MESZ schrieb Marc Roos :
>
>Hi Mehmet,
> 
>The data is already lost in these snapshots?

I cannot say that. Cause i did Not need the Snapshots. But you can try to Clone 
the vm in the state of the Snapshot ( i am using proxmox).  
  
> And how did you identify 
>the snapshot? It looks like I have these only in the rbd pool. 

You have to use "rbd info" to identify which Image is Related to this. Search 
for " 239f5274b0dc51"

- Mehmet 
>
>
>
>
>-Original Message-
>From: c...@elchaka.de [mailto:c...@elchaka.de] 
>Sent: zondag 8 april 2018 10:44
>To: ceph-users@lists.ceph.com
>Subject: Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for 
>$object?
>
>Hi Marc,
>
>Am 7. April 2018 18:32:40 MESZ schrieb Marc Roos 
>:
>>
>>How do you resolve these issues?
>>
>
>In my Case i could get rid of this by deleting the existing Snapshots.
>
>- Mehmet   
>>
>>Apr  7 22:39:21 c03 ceph-osd: 2018-04-07 22:39:21.928484 7f0826524700
>>-1
>>osd.13 pg_epoch: 19008 pg[17.13( v 19008'6019891 
>>(19008'6018375,19008'6019891] local-lis/les=18980/18981 n=3825
>>ec=3636/3636 lis/c 18980/18980 les/c/f 18981/18982/0
>18980/18980/18903)
>>
>>[4,13,0] r=1 lpr=18980 luod=0'0 crt=19008'6019891 lcod 19008'6019890 
>>active] _scan_snaps no head for
>>17:cbf61056:::rbd_data.239f5274b0dc51.0ff2:15 (have MIN) 
>>___
>>ceph-users mailing list
>>ceph-users@lists.ceph.com
>>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Marc Roos]

Hi Mehmet,
 
The data is already lost in these snapshots? And how did you identify 
the snapshot? It looks like I have these only in the rbd pool. 




-Original Message-
From: c...@elchaka.de [mailto:c...@elchaka.de] 
Sent: zondag 8 april 2018 10:44
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for 
$object?

Hi Marc,

Am 7. April 2018 18:32:40 MESZ schrieb Marc Roos 
:
>
>How do you resolve these issues?
>

In my Case i could get rid of this by deleting the existing Snapshots.

- Mehmet   
>
>Apr  7 22:39:21 c03 ceph-osd: 2018-04-07 22:39:21.928484 7f0826524700
>-1
>osd.13 pg_epoch: 19008 pg[17.13( v 19008'6019891 
>(19008'6018375,19008'6019891] local-lis/les=18980/18981 n=3825
>ec=3636/3636 lis/c 18980/18980 les/c/f 18981/18982/0 18980/18980/18903)
>
>[4,13,0] r=1 lpr=18980 luod=0'0 crt=19008'6019891 lcod 19008'6019890 
>active] _scan_snaps no head for
>17:cbf61056:::rbd_data.239f5274b0dc51.0ff2:15 (have MIN) 
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[ceph]

Hi Marc,

Am 7. April 2018 18:32:40 MESZ schrieb Marc Roos :
>
>How do you resolve these issues?
>

In my Case i could get rid of this by deleting the existing Snapshots.

- Mehmet   
>
>Apr  7 22:39:21 c03 ceph-osd: 2018-04-07 22:39:21.928484 7f0826524700
>-1 
>osd.13 pg_epoch: 19008 pg[17.13( v 19008'6019891 
>(19008'6018375,19008'6019891] local-lis/les=18980/18981 n=3825 
>ec=3636/3636 lis/c 18980/18980 les/c/f 18981/18982/0 18980/18980/18903)
>
>[4,13,0] r=1 lpr=18980 luod=0'0 crt=19008'6019891 lcod 19008'6019890 
>active] _scan_snaps no head for 
>17:cbf61056:::rbd_data.239f5274b0dc51.0ff2:15 (have MIN)
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com