[ceph-users] kernel:rbd:rbd0: encountered watch error: -10
Hi ! I meet a confused case: When write to cephfs and rbd at same time, after a while, rbd process is hang and i find: kernel:rbd:rbd0: encountered watch error: -10 I try to reproduce with below action and succeed: - run 2 dd process to write to cephfs - do file write action on rbd I find that lots of cpu are in iowait status, and lots of kernel process in D status. I guess that: - the process in the D state is mainly kswapd and writeback dirty page write-back thread process. when IO wait queue of the rbd disk is very long, then any process do IO operations on rbd disk, they need to be queued and wait for a long time and in the D state, the kernel will automatically print out the call stack after more than 120s - rbd hang since rbd client use watch-notify to communicate, when iowait stress is high, may do impact on it - cephfs and rbd share network bandwidth, and we use 40GB IB for ceph, network speed is too faster than disk speed Only workaround i can think about is refresh page cache by crond, but it may result in performance degradation. Could someone help me? Why rbd hang and how can I fix? I really want to use cephfs and rbd at same time, but this issue is so bad for production environment. Thanks ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] can not start osd service by systemd
Hi! I find a confused question about start/stop ceph cluster by systemd: - when cluster is on, restart ceph.target can restart all osd service - when cluster is down, start ceph.target or start ceph-osd.target can not start osd service I have google this issue, seems the workaround is start ceph-osd@n.service by hand. Is it a bug? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] [bug] mount.ceph man description is wrong
Sure. Seems that there is a test itself bug: https://jenkins.ceph.com/job/ceph-pull-requests-arm64/25498/console Best Wishes - Original Message - From: "Ilya Dryomov" To: "xiang.dai" Cc: "ceph-users" Sent: Wednesday, November 7, 2018 10:40:13 PM Subject: Re: [ceph-users] [bug] mount.ceph man description is wrong On Wed, Nov 7, 2018 at 2:25 PM wrote: > > Hi! > > I use ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic > (stable) and i want to call `ls -ld` to read whole dir size in cephfs: > > When i man mount.ceph: > > rbytes Report the recursive size of the directory contents for st_size on > directories. Default: on > > But without rbytes like below, "ls -ld" do not work: > > mount -t ceph 192.168.0.24:/ /mnt -o > name=admin,secretfile=/etc/ceph/admin.secret > > [root@test mnt]# ls -ld mongo > drwxr-xr-x 4 polkitd root 29 11月 6 16:33 mongo > > Then i umoun and mount use below cmd, it works: > > mount -t ceph 192.168.0.24:/ /mnt -o > name=admin,secretfile=/etc/ceph/admin.secret,rbytes > > > [root@test mnt]# ls -ld mongo > drwxr-xr-x 4 polkitd root 392021518 11月 6 16:33 mongo > > > So the description is wrong, right? Yes, it's wrong. Thanks for the PR, if you address the feedback we'll merge it. Ilya ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] [bug] mount.ceph man description is wrong
Hi! I use ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable) and i want to call `ls -ld` to read whole dir size in cephfs: When i man mount.ceph: rbytes Report the recursive size of the directory contents for st_size on directories. Default: on But without rbytes like below, "ls -ld" do not work: mount -t ceph 192.168.0.24:/ /mnt -o name=admin,secretfile=/etc/ceph/admin.secret [root@test mnt]# ls -ld mongo drwxr-xr-x 4 polkitd root 29 11月 6 16:33 mongo Then i umoun and mount use below cmd, it works: mount -t ceph 192.168.0.24:/ /mnt -o name=admin,secretfile=/etc/ceph/admin.secret,rbytes [root@test mnt]# ls -ld mongo drwxr-xr-x 4 polkitd root 392021518 11月 6 16:33 mongo So the description is wrong, right? Thanks ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] why set pg_num do not update pgp_num
Hi! I use ceph 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable), and find that: When expand whole cluster, i update pg_num, all succeed, but the status is as below: cluster: id: 41ef913c-2351-4794-b9ac-dd340e3fbc75 health: HEALTH_WARN 3 pools have pg_num > pgp_num Then i update pgp_num too, warning miss. What makes me confused is that when i create whole cluster at first time, i use "ceph osd create pool pool_name pg_num", the pgp_num is auto equal to pg_num. But "ceph osd set pool pool_name pg_num" not. Why does this design? Why do not auto update pgp_num when update pg_num? Thanks ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] how can i config pg_num
I install a ceph cluster with 8 osds, 3 pools and 1 replication(as osd_pool_default_size) in 2 machines. I follow formula in http://docs.ceph.com/docs/mimic/rados/operations/placement-groups/#choosing-the-number-of-placement-groups to count pg_nu. Then get total pg_num equal to 192, i set each pool as 64. I get below warn: $ ceph -s cluster: id: fd64d9e4-e33b-4e1c-927c-c0bb56d072cf health: HEALTH_WARN too few PGs per OSD (24 < min 30 ) Then i change osd_pool_size to 2, warning miss which makes me confused. I read docs again, i have below questions: 1.Between 5 and 10 OSDs set pg_num to 512 in doc, this pg_num is total pg num? If so, for 2 replications, the pg per osd is too low. If not, it means pg num per pool, for more pools, the pg per osd is too high. 2.How count the min is 30? 3.Why only change replication, the warning miss, seems that not count by the formula. 4.The formula does not consider pool num, just consider replication and osd num. So for more pool, the formula need to divide pool num too, right? 5.In http://docs.ceph.com/docs/mimic/rados/configuration/pool-pg-config-ref/, it says set 250 as default. This num is not power of 2, why set it? Is it right? If i set osd_pool_default_size as 2, does it mean need to set osd_pool_default_min_size as 1? If so, when osd_pool_default_size is 1, osd_pool_default_min_size equal to Zero? If not, for 2 machine: 1) set osd_pool_default_size as 2 is meaningless, but it can solve ceph status warning. 2) set osd_pool_default_size and osd_pool_default_min_size both 1? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph promethus monitor
Hi! I want to user promethus+grafana to monitor ceph, and I find below url: http: //docs.ceph.com/docs/master/mgr/prometheus/ Then i download ceph dashboard in grafana: https://grafana.com/dashboards/7056 It is so cool But some metrices do not work for ceph 13( Mimic ), like "ceph_monitor_latency_seconds" can not get now. On the other hand, some promethus query are in wrong syntax. Is there a repo to monitor and update dashborad and a repo to sync dashborad to `ceph module promethus`? Also, i want to contribute to them if possible. Thanks Xiang ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] issues about module promethus
Hi! I want to user promethus+grafana to monitor ceph, and I find below url: http: //docs.ceph.com/docs/master/mgr/prometheus/ Then i download ceph dashboard in grafana: https://grafana.com/dashboards/7056 It is so cool But some metrices do not work for ceph 13( Mimic ), like "ceph_monitor_latency_seconds" can not get now. On the other hand, some promethus query are in wrong syntax. Is there a repo to monitor and update dashborad and a repo to sync dashborad to `ceph module promethus`? Also, i want to contribute to them if possible. Thanks Xiang ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] issues about module promethus
Hi! I want to user promethus+grafana to monitor ceph, and I find below url: http: //docs.ceph.com/docs/master/mgr/prometheus/ Then i download ceph dashboard in grafana: https://grafana.com/dashboards/7056 It is so cool But some metrices do not work for ceph 13( Mimic ), like "ceph_monitor_latency_seconds" can not get now. Is there a repo to monitor and update dashborad and a repo to sync dashborad to `ceph module promethus`? Also, i want to contribute to them if possible. Thanks Xiang ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] stat file size is 0
I mount cephfs in /cephfs and create a dir in it: [root@test-0 guhailin]# ll -h drwxr-xr-x 1 guhailin ghlclass 0 8月 13 15:01 a scp a file into it: [root@test-0 guhailin]# ls a/ hadoop.tar.gz [root@test-0 guhailin]# pwd /cephfs/user/guhailin [root@test-0 guhailin]# stat a/ File: ‘a/’ Size: 0 Blocks: 0 IO Block: 65536 directory Device: 0h/0d Inode: 1099522168114 Links: 1 Access: (0755/drwxr-xr-x) Uid: ( 1018/guhailin) Gid: ( 1020/ghlclass) Access: 2018-08-13 15:00:29.060271071 +0800 Modify: 2018-08-13 15:01:20.669105642 +0800 Change: 2018-08-13 15:01:20.669105642 +0800 Birth: - After too much time, the size shows. I find if the dir is owned by root, no problem. [root@test-0 guhailin]# ceph --version ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable) So it is a bug of customized users? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] different size of rbd
I create a rbd named dx-app with 500G, and map as rbd0. But i find the size is different with different cmd: [root@dx-app docker]# rbd info dx-app rbd image 'dx-app': size 32000 GB in 8192000 objects < order 22 (4096 kB objects) block_name_prefix: rbd_data.1206643c9869 format: 2 features: layering flags: create_timestamp: Thu Aug 2 18:18:20 2018 [root@dx-app docker]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 20G 0 disk └─vda1 253:1 0 20G 0 part / vdb 253:16 0 200G 0 disk └─vg--test--data-lv--data 252:0 0 199.9G 0 lvm /test/data vdc 253:32 0 200G 0 disk vdd 253:48 0 200G 0 disk /pkgs vde 253:64 0 200G 0 disk rbd0 251:0 0 31.3T 0 disk /test/docker < [root@dx-app docker]# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/vda1 xfs 20G 14G 6.5G 68% / devtmpfs devtmpfs 7.8G 0 7.8G 0% /dev tmpfs tmpfs 7.8G 12K 7.8G 1% /dev/shm tmpfs tmpfs 7.8G 3.7M 7.8G 1% /run tmpfs tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup /dev/vde xfs 200G 33M 200G 1% /test/software /dev/vdd xfs 200G 117G 84G 59% /pkgs /dev/mapper/vg--test--data-lv--data xfs 200G 334M 200G 1% /test/data tmpfs tmpfs 1.6G 0 1.6G 0% /run/user/0 /dev/rbd0 xfs 500G 34M 500G 1% /test/docker < Which is true? Is it normal? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] qustions about rbdmap service
Hi! I find a rbd map service issue: [root@dx-test ~]# systemctl status rbdmap ● rbdmap.service - Map RBD devices Loaded: loaded (/usr/lib/systemd/system/rbdmap.service; enabled; vendor preset: disabled) Active: active (exited) (Result: exit-code) since 六 2018-07-28 13:55:01 CST; 11min ago Process: 1459 ExecStart=/usr/bin/rbdmap map (code=exited, status=1/FAILURE) Main PID: 1459 (code=exited, status=1/FAILURE) 7月 28 13:55:01 dx-test.novalocal systemd[1]: Started Map RBD devices. 7月 28 13:55:01 dx-test.novalocal systemd[1]: Starting Map RBD devices... 7月 28 14:01:19 dx-test.novalocal systemd[1]: rbdmap.service: main process exited, code=exited, status=1/FAILURE [root@dx-test ~]# echo $? 0 I am testing rbd map serive HA if ceph cluster down. I shut down ceph cluster and monitor rbdmap service, it spend 6 mins starting and failed. But it exits with 0 with $?. For HA, i remove oneshot Type and add Restart=always as below: [Unit] Description=Map RBD devices After=ceph.target network-online.target local-fs.target Wants=ceph.target network-online.target local-fs.target [Service] EnvironmentFile=-/etc/sysconfig/ceph Environment=RBDMAPFILE=/etc/ceph/rbdmap RemainAfterExit=yes ExecStart=/usr/bin/rbdmap map ExecReload=/usr/bin/rbdmap map ExecStop=/usr/bin/rbdmap unmap-all Restart=always RestartSec=3 [Install] WantedBy=multi-user.target But seems that Restart does not work. It still spends 6mins starting and failed, and when i start cluster, it also do nothing. Is it a design bug? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] questions about rbd used percentage
Hi!
I want to monitor rbd image size to enable enlager size when use percentage
above 80%.
I find a way with `rbd du`:
total=$(rbd du $rbd_name | grep $rbd_name | awk '{print $2}')
used=$(rbd du $rbd_name | grep $rbd_name | awk '{print $3}')
percentage=((used/total))
But in this way, it speed a lot time since fast-diff do not enable.
Is there a better way to check this?
Thanks
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ｒｂｄｍａｐ service issue
Hi! I find a rbd map service issue: [root@dx-test ~]# systemctl status rbdmap ● rbdmap.service - Map RBD devices Loaded: loaded (/usr/lib/systemd/system/rbdmap.service; enabled; vendor preset: disabled) Active: active (exited) (Result: exit-code) since 六 2018-07-28 13:55:01 CST; 11min ago Process: 1459 ExecStart=/usr/bin/rbdmap map (code=exited, status=1/FAILURE) Main PID: 1459 (code=exited, status=1/FAILURE) 7月 28 13:55:01 dx-test.novalocal systemd[1]: Started Map RBD devices. 7月 28 13:55:01 dx-test.novalocal systemd[1]: Starting Map RBD devices... 7月 28 14:01:19 dx-test.novalocal systemd[1]: rbdmap.service: main process exited, code=exited, status=1/FAILURE [root@dx-test ~]# echo $? 0 I am testing rbd map serive HA if ceph cluster down. I shut down ceph cluster and monitor rbdmap service, it spend 6 mins starting and failed. But it exits with 0 with $?. For HA, i remove oneshot Type and add Restart=always as below: [Unit] Description=Map RBD devices After=ceph.target network-online.target local-fs.target Wants=ceph.target network-online.target local-fs.target [Service] EnvironmentFile=-/etc/sysconfig/ceph Environment=RBDMAPFILE=/etc/ceph/rbdmap RemainAfterExit=yes ExecStart=/usr/bin/rbdmap map ExecReload=/usr/bin/rbdmap map ExecStop=/usr/bin/rbdmap unmap-all Restart=always RestartSec=3 [Install] WantedBy=multi-user.target But seems that Restart does not work. It still spends 6mins starting and failed, and when i start cluster, it also do nothing. Is it a design bug? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] rbdmap service failed but exit 1
Hi! I find a rbd map service issue: [root@dx-test ~]# systemctl status rbdmap ● rbdmap.service - Map RBD devices Loaded: loaded (/usr/lib/systemd/system/rbdmap.service; enabled; vendor preset: disabled) Active: active (exited) (Result: exit-code) since 六 2018-07-28 13:55:01 CST; 11min ago Process: 1459 ExecStart=/usr/bin/rbdmap map (code=exited, status=1/FAILURE) Main PID: 1459 (code=exited, status=1/FAILURE) 7月 28 13:55:01 dx-test.novalocal systemd[1]: Started Map RBD devices. 7月 28 13:55:01 dx-test.novalocal systemd[1]: Starting Map RBD devices... 7月 28 14:01:19 dx-test.novalocal systemd[1]: rbdmap.service: main process exited, code=exited, status=1/FAILURE [root@dx-test ~]# echo $? 0 I am testing rbd map serive HA if ceph cluster down. I shut down ceph cluster and monitor rbdmap service, it spend 6 mins starting and failed. But it exits with 0 with $?. For HA, i remove oneshot Type and add Restart=always as below: [Unit] Description=Map RBD devices After=ceph.target network-online.target local-fs.target Wants=ceph.target network-online.target local-fs.target [Service] EnvironmentFile=-/etc/sysconfig/ceph Environment=RBDMAPFILE=/etc/ceph/rbdmap RemainAfterExit=yes ExecStart=/usr/bin/rbdmap map ExecReload=/usr/bin/rbdmap map ExecStop=/usr/bin/rbdmap unmap-all Restart=always RestartSec=3 [Install] WantedBy=multi-user.target But seems that Restart does not work. It still spends 6mins starting and failed, and when i start cluster, it also do nothing. Is it a design bug? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] init mon fail since use service rather than systemctl
I met below issue:
INFO: initialize ceph mon ...
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.25): /usr/bin/ceph-deploy
--overwrite-conf mon create-initial
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts dx-storage
[ceph_deploy.mon][DEBUG ] detecting platform for host dx-storage ...
[dx-storage][DEBUG ] connected to host: dx-storage
[dx-storage][DEBUG ] detect platform information from remote host
[dx-storage][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: CentOS Linux 7.4.1708 Core
[dx-storage][DEBUG ] determining if provided host has same hostname in remote
[dx-storage][DEBUG ] get remote short hostname
[dx-storage][DEBUG ] deploying mon to dx-storage
[dx-storage][DEBUG ] get remote short hostname
[dx-storage][DEBUG ] remote hostname: dx-storage
[dx-storage][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[dx-storage][DEBUG ] create the mon path if it does not exist
[dx-storage][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-dx-storage/done
[dx-storage][DEBUG ] done path does not exist:
/var/lib/ceph/mon/ceph-dx-storage/done
[dx-storage][INFO ] creating keyring file:
/var/lib/ceph/tmp/ceph-dx-storage.mon.keyring
[dx-storage][DEBUG ] create the monitor keyring file
[dx-storage][INFO ] Running command: ceph-mon --cluster ceph --mkfs -i
dx-storage --keyring /var/lib/ceph/tmp/ceph-dx-storage.mon.keyring
[dx-storage][INFO ] unlinking keyring file
/var/lib/ceph/tmp/ceph-dx-storage.mon.keyring
[dx-storage][DEBUG ] create a done file to avoid re-doing the mon deployment
[dx-storage][DEBUG ] create the init path if it does not exist
[dx-storage][DEBUG ] locating the `service` executable...
[dx-storage][INFO ] Running command: /usr/sbin/service ceph -c
/etc/ceph/ceph.conf start mon.dx-storage
[dx-storage][WARNING] The service command supports only basic LSB actions
(start, stop, restart, try-restart, reload, force-reload, status). For other
actions, please try to use systemctl.
[dx-storage][ERROR ] RuntimeError: command returned non-zero exit status: 2
[ceph_deploy.mon][ERROR ] Failed to execute command: /usr/sbin/service ceph -c
/etc/ceph/ceph.conf start mon.dx-storage
[ceph_deploy][ERROR ] GenericError: Failed to create 1 monitors
My test host is centos7.4, and i think it should call systemctl rather than
service, but it still call service and failed.
My systemctl status is running, why ceph choose service rather than systemctl?
Could anyone tell me details?
Thanks in advance.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] how can i remove rbd0
Hi,all! I found a confused question: [root@test]# rbd ls [root@test]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 931.5G 0 disk ├─sda1 8:1 0 1G 0 part /boot ├─sda2 8:2 0 200G 0 part │ ├─root 253:0 0 50G 0 lvm / │ └─swap 253:1 0 8G 0 lvm [SWAP] └─sda3 8:3 0 186.3G 0 part sr0 11:0 1 1024M 0 rom rbd0 252:0 0 500G 0 disk <=== I have stopped rbdmap service. I do not want to reboot, how can i rm rbd0? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] which kernel support object-map, fast-diff
Hi, all! I use Centos 7.4 and want to use ceph rbd. I found that object-map, fast-diff can not work. rbd image 'app': size 500 GB in 128000 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.10a2643c9869 format: 2 features: layering, exclusive-lock, object-map, fast-diff <=== flags: object map invalid, fast diff invalid Ceph version is 12.2.2 (cf0baba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) Kernel: 3.10.0-693.el7.x86_64 So which kernel version support those feature? I do not find answer on ceph docs. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] rbd feature map fail
Hi, all! I use rbd to do something and find below issue: when i create a rbd image with feature: layering,exclusive-lock,object-map,fast-diff failed to map: rbd: sysfs write failed RBD image feature set mismatch. Try disabling features unsupported by the kernel with "rbd feature disable". In some cases useful info is found in syslog - try "dmesg | tail". rbd: map failed: (6) No such device or address dmesg | tail: [960284.869596] rbd: rbd0: capacity 107374182400 features 0x5 [960310.908615] libceph: mon1 10.0.10.12:6789 session established [960310.908916] libceph: client21459 fsid fe308030-ae94-471a-8d52-2c12151262fc [960310.911729] rbd: image foo: image uses unsupported features: 0x18 [960337.946856] libceph: mon1 10.0.10.12:6789 session established [960337.947320] libceph: client21465 fsid fe308030-ae94-471a-8d52-2c12151262fc [960337.950116] rbd: image foo: image uses unsupported features: 0x8 [960346.248676] libceph: mon0 10.0.10.11:6789 session established [960346.249077] libceph: client21866 fsid fe308030-ae94-471a-8d52-2c12151262fc [960346.254145] rbd: rbd0: capacity 107374182400 features 0x5 If i just create layering image, map is ok. *The question is here:* Then i enable feature: exclusive-lock,object-map,fast-diff It works. And rbd info shows all feature i set. I think it is a bug: why create with those feature then map failed but map after create is ok? I think it is more than order question. My OS is CentOS Linux release 7.4.1708 (Core), kernel is 3.10.0-693.el7.x86_64. Ceph version is 12.2.2 (cf0baba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) Thanks ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
