Re: [ceph-users] Ceph EC profile, how are you using?

2018-06-02 Thread Brady Deetz
One thing I'd love to see are benchmarks for ec profiles in different host/network configurations. We're building a new cluster and I will definitely be putting together an extensive benchmark panel together this time around so that we know exactly what works best in what situations. On Sat, Jun

Re: [ceph-users] Bug? ceph-volume zap not working

2018-06-02 Thread Marc Roos
If it creates it, should also remove it. Ceph-disk prepare did not create it, so it is logical if it does not remove it. ceph-volume however creates it, thus should remove it. -Original Message- From: David Turner [mailto:drakonst...@gmail.com] Sent: zaterdag 2 juni 2018 23:36 To:

Re: [ceph-users] Bug? ceph-volume zap not working

2018-06-02 Thread David Turner
Ceph-disk didn't remove an osd from the cluster either. That has never been a thing for ceph-disk or ceph-volume. There are other commands for that. On Sat, Jun 2, 2018, 4:29 PM Marc Roos wrote: > > But leaves still entries in crush map and maybe also ceph auth ls, and > the dir in

Re: [ceph-users] Bug? ceph-volume zap not working

2018-06-02 Thread Marc Roos
But leaves still entries in crush map and maybe also ceph auth ls, and the dir in /var/lib/ceph/osd -Original Message- From: Oliver Freyermuth [mailto:freyerm...@physik.uni-bonn.de] Sent: zaterdag 2 juni 2018 18:29 To: Marc Roos; ceph-users Subject: Re: [ceph-users] Bug? ceph-volume

Re: [ceph-users] Bug? Ceph-volume /var/lib/ceph/osd permissions

2018-06-02 Thread Oliver Freyermuth
Am 02.06.2018 um 12:35 schrieb Marc Roos: > > o+w? I don’t think that is necessary not? I also wondered about that, but it seems safe - it's only a tmpfs, with sticky bit set - and all files within have: -rw---. as you can check. Also, on our systems, we have: drwxr-x---. for /var/lib/ceph,

Re: [ceph-users] Should ceph-volume lvm prepare not be backwards compitable with ceph-disk?

2018-06-02 Thread Marc Roos
>> >> >> ceph-disk does not require bootstrap-osd/ceph.keyring and ceph-volume >> does > >I believe that's expected when you use "prepare". >For ceph-volume, "prepare" already bootstraps the OSD and fetches a fresh OSD id, for which it needs the keyring. >For ceph-disk, this was not

Re: [ceph-users] Should ceph-volume lvm prepare not be backwards compitable with ceph-disk?

2018-06-02 Thread Oliver Freyermuth
Am 02.06.2018 um 11:44 schrieb Marc Roos: > > > ceph-disk does not require bootstrap-osd/ceph.keyring and ceph-volume > does I believe that's expected when you use "prepare". For ceph-volume, "prepare" already bootstraps the OSD and fetches a fresh OSD id, for which it needs the keyring.

Re: [ceph-users] Bug? ceph-volume zap not working

2018-06-02 Thread Oliver Freyermuth
The command mapping from ceph-disk to ceph-volume is certainly not 1:1. What we are ended up using is: ceph-volume lvm zap /dev/sda --destroy This takes care of destroying Pvs and Lvs (as the documentation says). Cheers, Oliver Am 02.06.2018 um 12:16 schrieb Marc Roos: > > I guess zap

Re: [ceph-users] What is osd-lockbox? How does it work?

2018-06-02 Thread Jack
dmcrypt is part of the whole device-mapper infrastructure : https://en.wikipedia.org/wiki/Device_mapper LVM is nothing but a tool to manipulate device-mapper easily I do not see any reason for using dmcrypt directly without LVM On 06/02/2018 11:24 AM, Marc Roos wrote: > >>> I would like to try

[ceph-users] Bug? Ceph-volume /var/lib/ceph/osd permissions

2018-06-02 Thread Marc Roos
o+w? I don’t think that is necessary not? drwxr-xr-x 2 ceph ceph 182 May 9 12:59 ceph-15 drwxr-xr-x 2 ceph ceph 182 May 9 20:51 ceph-14 drwxr-xr-x 2 ceph ceph 182 May 12 10:32 ceph-16 drwxr-xr-x 2 ceph ceph 6 Jun 2 17:21 ceph-19 drwxr-x--- 13 ceph ceph 168 Jun 2 17:47 . drwxrwxrwt 2

Re: [ceph-users] Bug? ceph-volume zap not working

2018-06-02 Thread Marc Roos
After removing manually the lv, vg, and pv, ceph-volume zap works, but again the osd auth id still exists. (Although I have no idea how it should know which to remove) [@ bootstrap-osd]# ceph-volume lvm zap /dev/sdf --> Zapping: /dev/sdf Running command: /usr/sbin/cryptsetup status

[ceph-users] Bug? ceph-volume zap not working

2018-06-02 Thread Marc Roos
I guess zap should be used instead of destroy? Maybe keep ceph-disk backwards compatibility and keep destroy?? [root@c03 bootstrap-osd]# ceph-volume lvm zap /dev/sdf --> Zapping: /dev/sdf --> Unmounting /var/lib/ceph/osd/ceph-19 Running command: umount -v /var/lib/ceph/osd/ceph-19 stderr:

Re: [ceph-users] Data recovery after loosing all monitors

2018-06-02 Thread Bryan Henderson
> Kill all mds first , create new fs with old pools , then run ‘fs reset’ > before start any MDS. Brilliant! I can't wait to try it. Thanks. -- Bryan Henderson San Jose, California ___ ceph-users mailing list

[ceph-users] Bug? if ceph-volume fails, it does not clean up created osd auth id

2018-06-02 Thread Marc Roos
[@ bootstrap-osd]# ceph-volume lvm prepare --bluestore --data /dev/sdf Running command: /bin/ceph-authtool --gen-print-key Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c32036fe-ca0b-47d1-be3f-e28943ee3a97

[ceph-users] Should ceph-volume lvm prepare not be backwards compitable with ceph-disk?

2018-06-02 Thread Marc Roos
ceph-disk does not require bootstrap-osd/ceph.keyring and ceph-volume does [@~]# ceph-disk prepare --bluestore --zap-disk /dev/sdf *** Found invalid GPT and valid MBR; converting MBR to GPT format.

Re: [ceph-users] What is osd-lockbox? How does it work?

2018-06-02 Thread Marc Roos
>> I would like to try this disk encryption without lvm. > >> And then have ceph use this device dmcrypt device > >You know that dmcrypt and LVM are tightly linked, right ? No I did not. What makes this tightly linked relevant? ___ ceph-users mailing

Re: [ceph-users] What is osd-lockbox? How does it work?

2018-06-02 Thread Jack
On 06/02/2018 11:09 AM, Marc Roos wrote: > I would like to try this disk encryption without lvm. > And then have ceph use this device dmcrypt device You know that dmcrypt and LVM are tightly linked, right ? The later is a tool to handle the former

Re: [ceph-users] What is osd-lockbox? How does it work?

2018-06-02 Thread Marc Roos
Is this documented somewhere? Because I would like to try this disk encryption without lvm. I thought about just creating a regular bluestore disk, and then use dmcrypt on the 'ceph block' partition and use crypttab to automatically mount it. And then have ceph use this device dmcrypt

Re: [ceph-users] Ceph EC profile, how are you using?

2018-06-02 Thread Marc Roos
It would be nicer to keep such things on the mailinglist for future reference external links expire etc. -Original Message- From: Vasu Kulkarni [mailto:vakul...@redhat.com] Sent: vrijdag 1 juni 2018 18:51 To: ceph-users Subject: Re: [ceph-users] Ceph EC profile, how are you