Re: [ceph-users] ceph log level

[Marc Roos]

However, I can not get rid of these messages.

Dec 30 10:13:10 c02 ceph-mgr: 2019-12-30 10:13:10.343 7f7d3a2f8700  0 
log_channel(cluster) log [DBG] : pgmap v710220: 


-Original Message-
To: ceph-users; deaderzzs
Subject: Re: [ceph-users] ceph log level

I am decreasing logging with this script.


#!/bin/bash

declare -A logarrosd
declare -A logarrmon
declare -A logarrmgr

# default values luminous 12.2.7
logarrosd[debug_asok]="1/5"
logarrosd[debug_auth]="1/5"
logarrosd[debug_buffer]="0/1"
logarrosd[debug_client]="0/5"
logarrosd[debug_context]="0/1"
logarrosd[debug_crush]="1/1"
logarrosd[debug_filer]="0/1"
logarrosd[debug_filestore]="1/3"
logarrosd[debug_finisher]="1/1"
logarrosd[debug_heartbeatmap]="1/5"
logarrosd[debug_journal]="1/3"
logarrosd[debug_journaler]="0/5"
logarrosd[debug_lockdep]="0/1"
logarrosd[debug_mds]="1/5"
logarrosd[debug_mon]="1/5"
logarrosd[debug_monc]="0/10"
logarrosd[debug_ms]="0/5"
logarrosd[debug_objclass]="0/5"
logarrosd[debug_objectcacher]="0/5"
logarrosd[debug_objecter]="0/1"
logarrosd[debug_optracker]="0/5"
logarrosd[debug_osd]="1/5"
logarrosd[debug_paxos]="1/5"
logarrosd[debug_perfcounter]="1/5"
logarrosd[debug_rados]="0/5"
logarrosd[debug_rbd]="0/5"
logarrosd[debug_rgw]="1/5"
logarrosd[debug_rgw_sync]="1/5"
logarrosd[debug_throttle]="1/1"
logarrosd[debug_timer]="0/1"
logarrosd[debug_tp]="0/5"

logarrosd[debug_mds_balancer]="1/5"
logarrosd[debug_mds_locker]="1/5"
logarrosd[debug_mds_log]="1/5"
logarrosd[debug_mds_log_expire]="1/5"
logarrosd[debug_mds_migrator]="1/5"
logarrosd[debug_striper]="0/1"
logarrosd[debug_rbd_mirror]="0/5"
logarrosd[debug_rbd_replay]="0/5"
logarrosd[debug_crypto]="1/5"
logarrosd[debug_reserver]="1/1"
logarrosd[debug_civetweb]="1/10"
logarrosd[debug_javaclient]="1/5"
logarrosd[debug_xio]="1/5"
logarrosd[debug_compressor]="1/5"
logarrosd[debug_bluestore]="1/5"
logarrosd[debug_bluefs]="1/5" 
logarrosd[debug_bdev]="1/3"
logarrosd[debug_kstore]="1/5"
logarrosd[debug_rocksdb]="4/5"
logarrosd[debug_leveldb]="4/5"
logarrosd[debug_memdb]="4/5"
logarrosd[debug_kinetic]="1/5"
logarrosd[debug_fuse]="1/5"
logarrosd[debug_mgr]="1/5"
logarrosd[debug_mgrc]="1/5"
logarrosd[debug_dpdk]="1/5"
logarrosd[debug_eventtrace]="1/5"
logarrmon[debug_asok]="1/5"
logarrmon[debug_auth]="1/5"
logarrmon[debug_bdev]="1/3"
logarrmon[debug_bluefs]="1/5"
logarrmon[debug_bluestore]="1/5"
logarrmon[debug_buffer]="0/1"
logarrmon[debug_civetweb]="1/10"
logarrmon[debug_client]="0/5"
logarrmon[debug_compressor]="1/5"
logarrmon[debug_context]="0/1"
logarrmon[debug_crush]="1/1"
logarrmon[debug_crypto]="1/5"
logarrmon[debug_dpdk]="1/5"
logarrmon[debug_eventtrace]="1/5"
logarrmon[debug_filer]="0/1"
logarrmon[debug_filestore]="1/3"
logarrmon[debug_finisher]="1/1"
logarrmon[debug_fuse]="1/5"
logarrmon[debug_heartbeatmap]="1/5"
logarrmon[debug_javaclient]="1/5"
logarrmon[debug_journal]="1/3"
logarrmon[debug_journaler]="0/5"
logarrmon[debug_kinetic]="1/5"
logarrmon[debug_kstore]="1/5"
logarrmon[debug_leveldb]="4/5"
logarrmon[debug_lockdep]="0/1"
logarrmon[debug_mds]="1/5"
logarrmon[debug_mds_balancer]="1/5"
logarrmon[debug_mds_locker]="1/5"
logarrmon[debug_mds_log]="1/5"
logarrmon[debug_mds_log_expire]="1/5"
logarrmon[debug_mds_migrator]="1/5"
logarrmon[debug_memdb]="4/5"
logarrmon[debug_mgr]="1/5"
logarrmon[debug_mgrc]="1/5"
logarrmon[debug_mon]="1/5"
logarrmon[debug_monc]="0/10"
logarrmon[debug_ms]="0/0"
logarrmon[debug_none]="0/5"
logarrmon[debug_objclass]="0/5"
logarrmon[debug_objectcacher]="0/5"
logarrmon[debug_objecter]="0/1"
logarrmon[debug_optracker]="0/5"
logarrmon[debug_osd]="1/5"
logarrmon[debug_paxos]="1/5"
logarrmon[debug_perfcounter]="1/5"
logarrmon[debug_rados]="0/5"
logarrmon[debug_rbd]="0/5"
logarrmon[debug_rbd_mirror]="0/5"
logarrmon[debug_rbd_replay]="0/5"
logarrmon[debug_refs]="0/0"
logarrmon[debug_reserver]="1/1"
logarrmon[debug_rgw]="1/5"
logarrmon[debug_rgw_sync]="1/5"
logarrmon[debug_rocksdb]="4/5"
logarrmon[debug_striper]="0/1"
logarrmon[debug_throttle]="1/1"
logarrmon[debug_timer]="0/1"
logarrmon[debug_tp]="0/5"
logarrmon[debug_xio]="1/5"


for osdk in "${!logarrosd[@]}"
do
  ceph tell osd.* injectargs --$osdk=0/0 done

for monk in "${!logarrmon[@]}"
do
  ceph tell mon.* injectargs --$monk=0/0 done



-Original Message-
From: Zhenshi Zhou [mailto:deader...@gmail.com]
Sent: 30 December 2019 05:41
To: ceph-users
Subject: [ceph-users] ceph log level

Hi all,

OSD servers generate huge number of log. I configure 'debug_osd' to 1/5 
or 1/20, but it seems not working. Is there any other option which 
overrides this configuration?

Ceph version mimic(13.2.5)

Thanks


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph log level

[Marc Roos]

I am decreasing logging with this script.


#!/bin/bash

declare -A logarrosd
declare -A logarrmon
declare -A logarrmgr

# default values luminous 12.2.7
logarrosd[debug_asok]="1/5"
logarrosd[debug_auth]="1/5"
logarrosd[debug_buffer]="0/1"
logarrosd[debug_client]="0/5"
logarrosd[debug_context]="0/1"
logarrosd[debug_crush]="1/1"
logarrosd[debug_filer]="0/1"
logarrosd[debug_filestore]="1/3"
logarrosd[debug_finisher]="1/1"
logarrosd[debug_heartbeatmap]="1/5"
logarrosd[debug_journal]="1/3"
logarrosd[debug_journaler]="0/5"
logarrosd[debug_lockdep]="0/1"
logarrosd[debug_mds]="1/5"
logarrosd[debug_mon]="1/5"
logarrosd[debug_monc]="0/10"
logarrosd[debug_ms]="0/5"
logarrosd[debug_objclass]="0/5"
logarrosd[debug_objectcacher]="0/5"
logarrosd[debug_objecter]="0/1"
logarrosd[debug_optracker]="0/5"
logarrosd[debug_osd]="1/5"
logarrosd[debug_paxos]="1/5"
logarrosd[debug_perfcounter]="1/5"
logarrosd[debug_rados]="0/5"
logarrosd[debug_rbd]="0/5"
logarrosd[debug_rgw]="1/5"
logarrosd[debug_rgw_sync]="1/5"
logarrosd[debug_throttle]="1/1"
logarrosd[debug_timer]="0/1"
logarrosd[debug_tp]="0/5"

logarrosd[debug_mds_balancer]="1/5"
logarrosd[debug_mds_locker]="1/5"
logarrosd[debug_mds_log]="1/5"
logarrosd[debug_mds_log_expire]="1/5"
logarrosd[debug_mds_migrator]="1/5"
logarrosd[debug_striper]="0/1"
logarrosd[debug_rbd_mirror]="0/5"
logarrosd[debug_rbd_replay]="0/5"
logarrosd[debug_crypto]="1/5"
logarrosd[debug_reserver]="1/1"
logarrosd[debug_civetweb]="1/10"
logarrosd[debug_javaclient]="1/5"
logarrosd[debug_xio]="1/5"
logarrosd[debug_compressor]="1/5"
logarrosd[debug_bluestore]="1/5"
logarrosd[debug_bluefs]="1/5" 
logarrosd[debug_bdev]="1/3"
logarrosd[debug_kstore]="1/5"
logarrosd[debug_rocksdb]="4/5"
logarrosd[debug_leveldb]="4/5"
logarrosd[debug_memdb]="4/5"
logarrosd[debug_kinetic]="1/5"
logarrosd[debug_fuse]="1/5"
logarrosd[debug_mgr]="1/5"
logarrosd[debug_mgrc]="1/5"
logarrosd[debug_dpdk]="1/5"
logarrosd[debug_eventtrace]="1/5"
logarrmon[debug_asok]="1/5"
logarrmon[debug_auth]="1/5"
logarrmon[debug_bdev]="1/3"
logarrmon[debug_bluefs]="1/5"
logarrmon[debug_bluestore]="1/5"
logarrmon[debug_buffer]="0/1"
logarrmon[debug_civetweb]="1/10"
logarrmon[debug_client]="0/5"
logarrmon[debug_compressor]="1/5"
logarrmon[debug_context]="0/1"
logarrmon[debug_crush]="1/1"
logarrmon[debug_crypto]="1/5"
logarrmon[debug_dpdk]="1/5"
logarrmon[debug_eventtrace]="1/5"
logarrmon[debug_filer]="0/1"
logarrmon[debug_filestore]="1/3"
logarrmon[debug_finisher]="1/1"
logarrmon[debug_fuse]="1/5"
logarrmon[debug_heartbeatmap]="1/5"
logarrmon[debug_javaclient]="1/5"
logarrmon[debug_journal]="1/3"
logarrmon[debug_journaler]="0/5"
logarrmon[debug_kinetic]="1/5"
logarrmon[debug_kstore]="1/5"
logarrmon[debug_leveldb]="4/5"
logarrmon[debug_lockdep]="0/1"
logarrmon[debug_mds]="1/5"
logarrmon[debug_mds_balancer]="1/5"
logarrmon[debug_mds_locker]="1/5"
logarrmon[debug_mds_log]="1/5"
logarrmon[debug_mds_log_expire]="1/5"
logarrmon[debug_mds_migrator]="1/5"
logarrmon[debug_memdb]="4/5"
logarrmon[debug_mgr]="1/5"
logarrmon[debug_mgrc]="1/5"
logarrmon[debug_mon]="1/5"
logarrmon[debug_monc]="0/10"
logarrmon[debug_ms]="0/0"
logarrmon[debug_none]="0/5"
logarrmon[debug_objclass]="0/5"
logarrmon[debug_objectcacher]="0/5"
logarrmon[debug_objecter]="0/1"
logarrmon[debug_optracker]="0/5"
logarrmon[debug_osd]="1/5"
logarrmon[debug_paxos]="1/5"
logarrmon[debug_perfcounter]="1/5"
logarrmon[debug_rados]="0/5"
logarrmon[debug_rbd]="0/5"
logarrmon[debug_rbd_mirror]="0/5"
logarrmon[debug_rbd_replay]="0/5"
logarrmon[debug_refs]="0/0"
logarrmon[debug_reserver]="1/1"
logarrmon[debug_rgw]="1/5"
logarrmon[debug_rgw_sync]="1/5"
logarrmon[debug_rocksdb]="4/5"
logarrmon[debug_striper]="0/1"
logarrmon[debug_throttle]="1/1"
logarrmon[debug_timer]="0/1"
logarrmon[debug_tp]="0/5"
logarrmon[debug_xio]="1/5"


for osdk in "${!logarrosd[@]}"
do
  ceph tell osd.* injectargs --$osdk=0/0
done

for monk in "${!logarrmon[@]}"
do
  ceph tell mon.* injectargs --$monk=0/0
done



-Original Message-
From: Zhenshi Zhou [mailto:deader...@gmail.com] 
Sent: 30 December 2019 05:41
To: ceph-users
Subject: [ceph-users] ceph log level

Hi all,

OSD servers generate huge number of log. I configure 'debug_osd' to 1/5 
or 1/20, but it seems not working. Is there any other option which 
overrides this configuration?

Ceph version mimic(13.2.5)

Thanks


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] How can I stop this logging?

[Marc Roos]

 384 active+clean; 19 TiB data, 45 TiB used, 76 TiB / 122 TiB avail; 3.4 
KiB/s rd, 573 KiB/s wr, 20 op/s
Dec 23 11:58:25 c02 ceph-mgr: 2019-12-23 11:58:25.194 7f7d3a2f8700  0 
log_channel(cluster) log [DBG] : pgmap v411196: 384 pgs: 384 
active+clean; 19 TiB data, 45 TiB used, 76 TiB / 122 TiB avail; 3.3 
KiB/s rd, 521 KiB/s wr, 20 op/s
Dec 23 11:58:27 c02 ceph-mgr: 2019-12-23 11:58:27.196 7f7d3a2f8700  0 
log_channel(cluster) log [DBG] : pgmap v411197: 384 pgs: 384 
active+clean; 19 TiB data, 45 TiB used, 76 TiB / 122 TiB avail; 3.4 
KiB/s rd, 237 KiB/s wr, 19 op/s
Dec 23 11:58:29 c02 ceph-mgr: 2019-12-23 11:58:29.197 7f7d3a2f8700  0 
log_channel(cluster) log [DBG] : pgmap v411198: 384 pgs: 384 
active+clean; 19 TiB data, 45 TiB used, 76 TiB / 122 TiB avail; 3.2 
KiB/s rd, 254 KiB/s wr, 17 op/s
Dec 23 11:58:31 c02 ceph-mgr: 2019-12-23 11:58:31.199 7f7d3a2f8700  0 
log_channel(cluster) log [DBG] : pgmap v411199: 384 pgs: 384 
active+clean; 19 TiB data, 45 TiB used, 76 TiB / 122 TiB avail; 2.9 
KiB/s rd, 258 KiB/s wr, 17 op/s
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Separate disk sets for high IO?

[Marc Roos]

You can classify osd's, eg as ssd. And you can assign this class to a 
pool you create. This way you have have rbd's running on only ssd's. I 
think you have also a class for nvme and you can create custom classes.


 

-Original Message-
From: Philip Brown [mailto:pbr...@medata.com] 
Sent: 16 December 2019 22:55
To: ceph-users
Subject: [ceph-users] Separate disk sets for high IO?

Still relatively new to ceph, but have been tinkering for a few weeks 
now.

If I'm reading the various docs correctly, then any RBD in a particular 
ceph cluster, will be distributed across ALL OSDs, ALL the time.
There is no way to designate a particular set of disks, AKA OSDs, to be 
a high performance group, and allocate certain RBDs to only use that set 
of disks.
Pools, only control things like the replication count, and number of 
placement groups.

I'd have to set up a whole new ceph cluster for the type of behavior I 
want.

Am I correct?



--
Philip Brown| Sr. Linux System Administrator | Medata, Inc. 
5 Peters Canyon Rd Suite 250
Irvine CA 92606
Office 714.918.1310| Fax 714.918.1325
pbr...@medata.com| www.medata.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] deleted snap dirs are back as _origdir_1099536400705

[Marc Roos]

Yes Thanks!!! you are right I deleted the higher created snapshots, and 
they are now gone.
 

-Original Message-
Cc: ceph-users
Subject: Re: [ceph-users] deleted snap dirs are back as 
_origdir_1099536400705

With just the one ls listing and my memory it's not totally clear, but I 
believe this is the output you get when delete a snapshot folder but 
it's still referenced by a different snapshot farther up the hierarchy.
-Greg

On Mon, Dec 16, 2019 at 8:51 AM Marc Roos  
wrote:
>
>
> Am I the only lucky one having this problem? Should I use the 
> bugtracker system for this?
>
> -Original Message-
> From: Marc Roos
> Sent: 14 December 2019 10:05
> Cc: ceph-users
> Subject: Re: [ceph-users] deleted snap dirs are back as
> _origdir_1099536400705
>
>
>
> ceph tell mds.a scrub start / recursive repair Did not fix this.
>
>
>
> -Original Message-
> Cc: ceph-users
> Subject: [ceph-users] deleted snap dirs are back as
> _origdir_1099536400705
>
>
> I thought I deleted snapshot dirs, but I still have them but with a 
> different name. How to get rid of these?
>
> [@ .snap]# ls -1
> _snap-1_1099536400705
> _snap-2_1099536400705
> _snap-3_1099536400705
> _snap-4_1099536400705
> _snap-5_1099536400705
> _snap-6_1099536400705
> _snap-7_1099536400705
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] deleted snap dirs are back as _origdir_1099536400705

[Marc Roos]

 
Am I the only lucky one having this problem? Should I use the bugtracker 
system for this?

-Original Message-
From: Marc Roos 
Sent: 14 December 2019 10:05
Cc: ceph-users
Subject: Re: [ceph-users] deleted snap dirs are back as 
_origdir_1099536400705

 

ceph tell mds.a scrub start / recursive repair Did not fix this.



-Original Message-
Cc: ceph-users
Subject: [ceph-users] deleted snap dirs are back as
_origdir_1099536400705

 
I thought I deleted snapshot dirs, but I still have them but with a 
different name. How to get rid of these?

[@ .snap]# ls -1
_snap-1_1099536400705
_snap-2_1099536400705
_snap-3_1099536400705
_snap-4_1099536400705
_snap-5_1099536400705
_snap-6_1099536400705
_snap-7_1099536400705

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] deleted snap dirs are back as _origdir_1099536400705

[Marc Roos]

 

ceph tell mds.a scrub start / recursive repair
Did not fix this.



-Original Message-
Cc: ceph-users
Subject: [ceph-users] deleted snap dirs are back as 
_origdir_1099536400705

 
I thought I deleted snapshot dirs, but I still have them but with a 
different name. How to get rid of these?

[@ .snap]# ls -1
_snap-1_1099536400705
_snap-2_1099536400705
_snap-3_1099536400705
_snap-4_1099536400705
_snap-5_1099536400705
_snap-6_1099536400705
_snap-7_1099536400705
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph tell mds.a scrub status "problem getting command descriptions"

[Marc Roos]

 
client.admin, did not have correct rights

 ceph auth caps client.admin mds "allow *" mgr "allow *" mon "allow *" 
osd "allow *"


-Original Message-
To: ceph-users
Subject: [ceph-users] ceph tell mds.a scrub status "problem getting 
command descriptions"


ceph tell mds.a scrub status

Generates

2019-12-14 00:46:38.782 7fef4affd700  0 client.3744774 ms_handle_reset 
on v2:192.168.10.111:6800/3517983549 Error EPERM: problem getting 
command descriptions from mds.a 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph tell mds.a scrub status "problem getting command descriptions"

[Marc Roos]

ceph tell mds.a scrub status

Generates

2019-12-14 00:46:38.782 7fef4affd700  0 client.3744774 ms_handle_reset 
on v2:192.168.10.111:6800/3517983549
Error EPERM: problem getting command descriptions from mds.a
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] deleted snap dirs are back as _origdir_1099536400705

[Marc Roos]

 
I thought I deleted snapshot dirs, but I still have them but with a 
different name. How to get rid of these?

[@ .snap]# ls -1
_snap-1_1099536400705
_snap-2_1099536400705
_snap-3_1099536400705
_snap-4_1099536400705
_snap-5_1099536400705
_snap-6_1099536400705
_snap-7_1099536400705
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] sharing single SSD across multiple HD based OSDs

[Marc Roos]

Just also a bit curious. So it just creates a pv on sda and no 
partitioning done on sda? 


-Original Message-
From: Daniel Sung [mailto:daniel.s...@quadraturecapital.com] 
Sent: dinsdag 10 december 2019 14:40
To: Philip Brown
Cc: ceph-users
Subject: Re: [ceph-users] sharing single SSD across multiple HD based 
OSDs

It just uses LVM to create a bunch of LVs. It doesn't actually create 
separate partitions on the block devices. You can run the command and it 
will give you a preview of what it will do and ask for confirmation. 

On Tue, 10 Dec 2019 at 13:36, Philip Brown  wrote:


Interesting. What did the partitioning look like?

- Original Message -
From: "Daniel Sung" 
To: "Nathan Fish" 
Cc: "Philip Brown" , "ceph-users" 

Sent: Tuesday, December 10, 2019 1:21:36 AM
Subject: Re: [ceph-users] sharing single SSD across multiple HD 
based OSDs

The way I did this was to use:

ceph-volume lvm batch /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde
/dev/sdf etc

Where you just list all of the block devices you want to use in a 
group. It
will automatically determine which devices are SSD and then 
automatically
partition it for you and share it amongst the HDD OSDs.




-- 

Quadrature Capital
daniel.s...@quadraturecapital.com
http://www.quadraturecapital.com
Dir: +44-20-3743-0428 Main: +44-20-3743-0400 Fax: +44-20-3743-0401 The 
Leadenhall Building, 122 Leadenhall Street, London, EC3V 4AB


---
Quadrature Capital Limited, a limited company, registered in England and 
Wales with registered number 09516131, is authorised and regulated by 
the Financial Conduct Authority.

Any e-mail sent from Quadrature Capital may contain information which is 
confidential. Unless you are the intended recipient, you may not 
disclose, copy or use it; please notify the sender immediately and 
delete it and any copies from your systems. You should protect your 
system from viruses etc.; we accept no responsibility for damage that 
may be caused by them.

We may monitor email content for the purposes of ensuring compliance 
with law and our policies, as well as details of correspondents to 
supplement our relationships database.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph-mgr :: Grafana + Telegraf / InfluxDB metrics format

[Marc Roos]

 >I am having hard times with creating graphs I want to see. Metrics are 
exported in way that every single one is stored in separated series in 
Influx like:
 >
 >> ceph_pool_stats,cluster=ceph1,metric=read value=1234 
15506589110
 >> ceph_pool_stats,cluster=ceph1,metric=write value=1234 
15506589110
 >> ceph_pool_stats,cluster=ceph1,metric=total value=1234 
15506589110
 >
 >instead of single series like:
 >
 >> ceph_pool_stats,cluster=ceph1 read=1234,write=1234,total=1234 
15506589110
 >

That is how timeseries databases work, one value per timestamp

 >This means when I want to create graph of something like % usage ratio 
(= bytes_used / bytes_total) or number of faulty OSDs (= num_osd_up - 
num_osd_in) I am unable to do it with single query like

I use a static variable in influx to fix this. I have static value for 
osds and totalbytes (although newer ceph versions log this i think). You 
can also solve this with continuous queries in influx. 

 >> SELECT mean("num_osd_up") - mean("num_osd_in") FROM 
"ceph_cluster_stats" WHERE "cluster" =~ /^ceph1$/ AND time >= now() - 6h 
GROUP BY time(5m) fill(null)
 
SELECT $totalosds - last("value")  FROM "ceph_value" WHERE "type" = 
'ceph_bytes' AND "type_instance" = 'Cluster.numOsdIn' AND $timeFilter 
GROUP BY time($__interval) fill(null)
 
 >but instead it requires two queries followed by math operation, which 
I was unable to get it working in my Grafana nor InfluxDB (I believe 
it's not supported, Influx removed JOIN queries some time ago).
 > 



-Original Message-
From: Miroslav Kalina [mailto:miroslav.kal...@livesport.eu] 
Sent: dinsdag 10 december 2019 13:31
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Ceph-mgr :: Grafana + Telegraf / InfluxDB metrics 
format

Hello guys,

is there anyone using Telegraf / InfluxDB metrics exporter with Grafana 
dashboards? I am asking like that because I was unable to find any 
existing Grafana dashboards based on InfluxDB.

I am having hard times with creating graphs I want to see. Metrics are 
exported in way that every single one is stored in separated series in 
Influx like:

> ceph_pool_stats,cluster=ceph1,metric=read value=1234 
15506589110
> ceph_pool_stats,cluster=ceph1,metric=write value=1234 
15506589110
> ceph_pool_stats,cluster=ceph1,metric=total value=1234 
15506589110

instead of single series like:

> ceph_pool_stats,cluster=ceph1 read=1234,write=1234,total=1234 
15506589110

This means when I want to create graph of something like % usage ratio 
(= bytes_used / bytes_total) or number of faulty OSDs (= num_osd_up - 
num_osd_in) I am unable to do it with single query like

> SELECT mean("num_osd_up") - mean("num_osd_in") FROM 
"ceph_cluster_stats" WHERE "cluster" =~ /^ceph1$/ AND time >= now() - 6h 
GROUP BY time(5m) fill(null)

but instead it requires two queries followed by math operation, which I 
was unable to get it working in my Grafana nor InfluxDB (I believe it's 
not supported, Influx removed JOIN queries some time ago).

I didn't see any possibility how to modify metrics format exported to 
Telegraf. I feel like I am missing something pretty obvious here.

I am currently unable to switch to prometheus exporter (which don't have 
this kind of issue) because of my current infrastructure setup.

Currently I am using following versions:
 * Ceph 14.2.4
 * InfluxDB 1.6.4
 * Grafana 6.4.2

So ... do you have it working anyone? Please could you share your 
dashboards?

Best regards

-- 
Miroslav Kalina
Systems developement specialist

Livesport s.r.o.
Aspira Business Centre
Bucharova 2928/14a, 158 00 Praha 5
www.livesport.eu


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Qemu RBD image usage

[Marc Roos]

 
This should get you started with using rbd.


  
  

  
  



  
  
  WDC
  WD40EFRX-68WT0N0
  
  
  





cat > secret.xml <


client.rbd.vps secret


EOF

virsh secret-define --file secret.xml

virsh secret-set-value --secret  --base64 `ceph auth get-key 
client.rbd.vps 2>/dev/null`



-Original Message-
To: ceph-users@lists.ceph.com
Cc: d...@ceph.io
Subject: [ceph-users] Qemu RBD image usage

Hi all,
   I want to attach another RBD image into the Qemu VM to be used as 
disk.
   However, it always failed.  The VM definiation xml is attached.
   Could anyone tell me where I did wrong?
   || nstcc3@nstcloudcc3:~$ sudo virsh start ubuntu_18_04_mysql 
--console
   || error: Failed to start domain ubuntu_18_04_mysql
   || error: internal error: process exited while connecting to monitor:
   || 2019-12-09T16:24:30.284454Z qemu-system-x86_64: -drive
   || 
file=rbd:rwl_mysql/mysql_image:auth_supported=none:mon_host=nstcloudcc4\
:6789,format=raw,if=none,id=drive-virtio-disk1:
   || error connecting: Operation not supported


   The cluster info is below:
   || ceph@nstcloudcc3:~$ ceph --version
   || ceph version 14.0.0-16935-g9b6ef711f3 
(9b6ef711f3a40898756457cb287bf291f45943f0) octopus (dev)
   || ceph@nstcloudcc3:~$ ceph -s
   ||   cluster:
   || id: e31502ff-1fb4-40b7-89a8-2b85a77a3b09
   || health: HEALTH_OK
   ||  
   ||   services:
   || mon: 1 daemons, quorum nstcloudcc4 (age 2h)
   || mgr: nstcloudcc4(active, since 2h)
   || osd: 4 osds: 4 up (since 2h), 4 in (since 2h)
   ||  
   ||   data:
   || pools:   1 pools, 128 pgs
   || objects: 6 objects, 6.3 KiB
   || usage:   4.0 GiB used, 7.3 TiB / 7.3 TiB avail
   || pgs: 128 active+clean
   ||  
   || ceph@nstcloudcc3:~$
   || ceph@nstcloudcc3:~$ rbd info rwl_mysql/mysql_image
   || rbd image 'mysql_image':
   || size 100 GiB in 25600 objects
   || order 22 (4 MiB objects)
   || snapshot_count: 0
   || id: 110feda39b1c
   || block_name_prefix: rbd_data.110feda39b1c
   || format: 2
   || features: layering, exclusive-lock, object-map, fast-diff, 
deep-flatten
   || op_features: 
   || flags: 
   || create_timestamp: Mon Dec  9 23:48:17 2019
   || access_timestamp: Mon Dec  9 23:48:17 2019
   || modify_timestamp: Mon Dec  9 23:48:17 2019

B.R.
Changcheng


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 2 different ceph-users lists?

[Marc Roos]

 

ceph-users@lists.ceph.com is old one, why this is, I also do not know

https://www.mail-archive.com/search?l=all=ceph


-Original Message-
From: Rodrigo Severo - Fábrica [mailto:rodr...@fabricadeideias.com] 
Sent: donderdag 5 december 2019 20:37
To: ceph-users@lists.ceph.com; ceph-us...@ceph.io
Subject: [ceph-users] 2 different ceph-users lists?

Hi,


Are there 2 different ceph-users list?

ceph-users@lists.ceph.com

and

ceph-us...@ceph.io

Why? What's the difference?


Regards,

Rodrigo Severo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] SSDs behind Hardware Raid

[Marc Roos]

 
But I guess that in 'ceph osd tree' the ssd's were then also displayed 
as hdd?



-Original Message-
From: Stolte, Felix [mailto:f.sto...@fz-juelich.de] 
Sent: woensdag 4 december 2019 9:12
To: ceph-users
Subject: [ceph-users] SSDs behind Hardware Raid

Hi guys,

 

maybe this is common knowledge for the most of you, for me it was not:

 

if you are using SSDs behind a raid controller in raid mode  (not JBOD) 
make sure your operating system treats them correctly as SSDs. I am an 
Ubuntu user but I think the following applies to all linux operating 
systems:

 

/sys/block//queue/rotational determines if an device is 
treated as rotational or not. 0 stands for SSD, 1 for Rotational.

 

In my case Ubuntu treated my SSDs (Raid 0, 1 Disk) as rotational. 
Changing the parameter above for my SSDs to 0 and restarting the 
corresponding osd daemons increased 4K write IOPS drastically:

 

rados -p ssd bench 60 write -b 4K (6 Nodes, 3 SSDs each)

 

Before: ~5200 IOPS

After: ~11500 IOPS

 

@Developers: I am aware that this is not directly a ceph issue, but 
maybe you could consider to add this hint in your documentation. I could 
be wrong, but I think I am not the only one using a hw raid controller 
for osds (not willingly by the way).

 

On a sidenote: ceph-volume lvm batch uses the rotational parameter as 
well for identifying SSDs (please correct me if I am wrong).

 

Best regards

Felix

 


-


-

Forschungszentrum Juelich GmbH

52425 Juelich

Sitz der Gesellschaft: Juelich

Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498

Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher

Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),

Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,

Prof. Dr. Sebastian M. Schmidt


-


-

 


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd image size

[Marc Roos]

Is there a point to sending such signature (twice) to a public mailing 
list, having its emails stored on serveral mailing list websites?









===
PS don't you think this (^) is a nicer line than





/

-Original Message-
To: ceph-users
Subject: [ceph-users] rbd image size

Hello ,  I  use ceph as block storage in kubernetes. I want to get the 
rbd usage by command "rbd diff image_id | awk '{ SUM += $2 } END { print 
SUM/1024/1024 " MB" }’”, but I found it is a lot bigger than the value 
which I got by command “df -h” in the pod. I do not know the reason 
and need your help.

Thanks.




// 
声明：此邮件可能包含依图公司保密或特权信息，并且仅应发送至有权接收该邮件
的收件人。如果您无权收取该邮件，您应当立即删除该邮件并通知发件人，您并被
禁止传播、分发或复制此邮件以及附件。对于此邮件可能携带的病毒引起的任何损
害，本公司不承担任何责任。此外，本公司不保证已正确和完整地传输此信息，也
不接受任何延迟收件的赔偿责任。 




// Notice: 
This email may contain confidential or privileged information of Yitu 
and was sent solely to the intended recipients. If you are unauthorized 
to receive this email, you should delete the email and contact the 
sender immediately. Any unauthorized disclosing, distribution, or 
copying of this email and attachment thereto is prohibited. Yitu does 
not accept any liability for any loss caused by possibly viruses in this 
email. E-mail transmission cannot be guaranteed to be secure or 
error-free and Yitu is not responsible for any delayed transmission.




// 
声明：此邮件可能包含依图公司保密或特权信息，并且仅应发送至有权接收该邮件
的收件人。如果您无权收取该邮件，您应当立即删除该邮件并通知发件人，您并被
禁止传播、分发或复制此邮件以及附件。对于此邮件可能携带的病毒引起的任何损
害，本公司不承担任何责任。此外，本公司不保证已正确和完整地传输此信息，也
不接受任何延迟收件的赔偿责任。 




// Notice: 
This email may contain confidential or privileged information of Yitu 
and was sent solely to the intended recipients. If you are unauthorized 
to receive this email, you should delete the email and contact the 
sender immediately. Any unauthorized disclosing, distribution, or 
copying of this email and attachment thereto is prohibited. Yitu does 
not accept any liability for any loss caused by possibly viruses in this 
email. E-mail transmission cannot be guaranteed to be secure or 
error-free and Yitu is not responsible for any delayed transmission.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cloudstack and CEPH Day London

[Marc Roos]

 
I was thinking of going to the Polish one, but I will be tempted to go 
to the London one, if you also be wearing this Kilt. ;D


-Original Message-
From: John Hearns [mailto:hear...@googlemail.com] 
Sent: donderdag 24 oktober 2019 8:14
To: ceph-users
Subject: [ceph-users] Cloudstack and CEPH Day London

I will be attending the Cloudstack and CEPH Day in London today.
Please say hello - rotund Scottish guy, not much hair. Glaswegian 
accent!


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] hanging slow requests: failed to authpin, subtree is being exported

[Marc Roos]

 
I think I am having this issue also (at least I had with luminous) I had 
to remove the hidden temp files rsync had left, when the cephfs mount 
'stalled', otherwise I would never be able to complete the rsync.


-Original Message-
Cc: ceph-users
Subject: Re: [ceph-users] hanging slow requests: failed to authpin, 
subtree is being exported


I've made a ticket for this issue: https://tracker.ceph.com/issues/42338

Thanks again!

K

On 15/10/2019 18:00, Kenneth Waegeman wrote:
> Hi Robert, all,
>
>
> On 23/09/2019 17:37, Robert LeBlanc wrote:
>> On Mon, Sep 23, 2019 at 4:14 AM Kenneth Waegeman 
>>  wrote:
>>> Hi all,
>>>
>>> When syncing data with rsync, I'm often getting blocked slow 
>>> requests, which also block access to this path.
>>>
 2019-09-23 11:25:49.477 7f4f401e8700 0 log_channel(cluster) log 
 [WRN]
 : slow request 31.895478 seconds old, received at 2019-09-23
 11:25:17.598152: client_request(client.38352684:92684 lookup
 #0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
 caller_gid=0{0,}) currently failed to authpin, subtree is being 
 exported
 2019-09-23 11:26:19.477 7f4f401e8700  0 log_channel(cluster) log 
 [WRN]
 : slow request 61.896079 seconds old, received at 2019-09-23
 11:25:17.598152: client_request(client.38352684:92684 lookup
 #0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
 caller_gid=0{0,}) currently failed to authpin, subtree is being 
 exported
 2019-09-23 11:27:19.478 7f4f401e8700  0 log_channel(cluster) log 
 [WRN]
 : slow request 121.897268 seconds old, received at 2019-09-23
 11:25:17.598152: client_request(client.38352684:92684 lookup
 #0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
 caller_gid=0{0,}) currently failed to authpin, subtree is being 
 exported
 2019-09-23 11:29:19.488 7f4f401e8700  0 log_channel(cluster) log 
 [WRN]
 : slow request 241.899467 seconds old, received at 2019-09-23
 11:25:17.598152: client_request(client.38352684:92684 lookup
 #0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
 caller_gid=0{0,}) currently failed to authpin, subtree is being 
 exported
 2019-09-23 11:33:19.680 7f4f401e8700  0 log_channel(cluster) log 
 [WRN]
 : slow request 482.087927 seconds old, received at 2019-09-23
 11:25:17.598152: client_request(client.38352684:92684 lookup
 #0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
 caller_gid=0{0,}) currently failed to authpin, subtree is being 
 exported
 2019-09-23 11:36:09.881 7f4f401e8700  0 log_channel(cluster) log 
 [WRN]
 : slow request 32.677511 seconds old, received at 2019-09-23
 11:35:37.217113: client_request(client.38347357:111963 lookup 
 #0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
 caller_gid=0{0,}) currently failed to authpin, subtree is being 
 exported
 2019-09-23 11:36:39.881 7f4f401e8700  0 log_channel(cluster) log 
 [WRN]
 : slow request 62.678132 seconds old, received at 2019-09-23
 11:35:37.217113: client_request(client.38347357:111963 lookup 
 #0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
 caller_gid=0{0,}) currently failed to authpin, subtree is being 
 exported
 2019-09-23 11:37:39.891 7f4f401e8700  0 log_channel(cluster) log 
 [WRN]
 : slow request 122.679273 seconds old, received at 2019-09-23
 11:35:37.217113: client_request(client.38347357:111963 lookup 
 #0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
 caller_gid=0{0,}) currently failed to authpin, subtree is being 
 exported
 2019-09-23 11:39:39.892 7f4f401e8700  0 log_channel(cluster) log 
 [WRN]
 : slow request 242.684667 seconds old, received at 2019-09-23
 11:35:37.217113: client_request(client.38347357:111963 lookup 
 #0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
 caller_gid=0{0,}) currently failed to authpin, subtree is being 
 exported
 2019-09-23 11:41:19.893 7f4f401e8700  0 log_channel(cluster) log 
 [WRN]
 : slow request 962.305681 seconds old, received at 2019-09-23
 11:25:17.598152: client_request(client.38352684:92684 lookup
 #0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
 caller_gid=0{0,}) currently failed to authpin, subtree is being 
 exported
 2019-09-23 11:43:39.923 7f4f401e8700  0 log_channel(cluster) log 
 [WRN]
 : slow request 482.712888 seconds old, received at 2019-09-23
 11:35:37.217113: client_request(client.38347357:111963 lookup 
 #0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
 caller_gid=0{0,}) currently failed to authpin, subtree is being 
 exported
 2019-09-23 11:51:40.236 7f4f401e8700  0 log_channel(cluster) log 
 [WRN]
 : slow request 963.037049 seconds old, received at 2019-09-23
 11:35:37.217113: 

Re: [ceph-users] collectd Ceph metric

[Marc Roos]

 
The 'xx-.conf' are mine, custom. So I would not have to merge 
changes with newer /etc/collectd.conf rpm updates. 

I would suggest get a small configuration that is working, set debug 
logging[0], and increase the configuration until it fails with little 
steps. Load plugin ceph empty, configure then one osd, fails? try mgr?, 
try mon? etc


[0] Try something like this?
LoadPlugin logfile


#not compiled with debug?
#also not writing to the logfile
LogLevel debug
 File "/tmp/collectd.log"
#File STDOUT
Timestamp true
#   PrintSeverity false



-Original Message-
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] collectd Ceph metric

Is there any instruction to install the plugin configuration?

Attach my RHEL/collectd configuration file under /etc/ directory.
On RHEL:
[rdma@rdmarhel0 collectd.d]$ pwd
/etc/collectd.d
[rdma@rdmarhel0 collectd.d]$ tree .
.

0 directories, 0 files
[rdma@rdmarhel0 collectd.d]$

I've also checked the collectd configuration file under Ubuntu, there's 
no 11-network.conf or 12-memory.conf .etc. However, it could still 
collect the cpu and memory information.
On Ubuntu:
   nstcc1@nstcloudcc1:collectd$ pwd
   /etc/collectd
   nstcc1@nstcloudcc1:collectd$ tree .
   .
   ├── collectd.conf
   ├── collectd.conf.d
   │   ├── filters.conf
   │   └── thresholds.conf
   └── collection.conf

   1 directory, 4 files
   nstcc1@nstcloudcc1:collectd$

B.R.
Changcheng

On 10:56 Mon 21 Oct, Marc Roos wrote:
> 
> Your collectd starts without the ceph plugin ok? 
> 
> I have also your error " didn't register a configuration callback", 
> because I configured debug logging, but did not enable it by loading 
> the plugin 'logfile'. Maybe it is the order in which your 
> configuration files a read (I think this used to be important with 
> collectd)
> 
> I have only in my collectd.conf these two lines:
> Include "/etc/collectd.d"
> LoadPlugin syslog
> 
> And in /etc/collectd.d/ these files:
> 10-custom.conf(with network section for influx)
> 11-network.conf   (ethx)
> 12-memory.conf
> 50-ceph.conf
> 51-ipmi.conf
> 52-ipmi-power.conf
> 53-disk.conf
> 
> 
> 
> 
> [0] journalctl -u collectd.service
> Oct 21 10:29:39 c02 collectd[1017750]: Exiting normally.
> Oct 21 10:29:39 c02 systemd[1]: Stopping Collectd statistics daemon...
> Oct 21 10:29:39 c02 collectd[1017750]: collectd: Stopping 5 read 
> threads.
> Oct 21 10:29:39 c02 collectd[1017750]: collectd: Stopping 5 write 
> threads.
> Oct 21 10:29:40 c02 systemd[1]: Stopped Collectd statistics daemon.
> Oct 21 10:29:53 c02 systemd[1]: Starting Collectd statistics daemon...
> Oct 21 10:29:53 c02 collectd[1031939]: plugin_load: plugin "syslog" 
> successfully loaded.
> Oct 21 10:29:53 c02 collectd[1031939]: plugin_load: plugin "network" 
> successfully loaded.
> Oct 21 10:29:53 c02 collectd[1031939]: Found a configuration for the 
> `logfile' plugin, but the plugin isn't loaded or didn't register a 
> configuration callback.
> Oct 21 10:29:53 c02 collectd[1031939]: Found a configuration for the 
> `logfile' plugin, but the plugin isn't loaded or didn't register a 
> configuration callback.
> Oct 21 10:29:53 c02 collectd[1031939]: Found a configuration for the 
> `logfile' plugin, but the plugin isn't loaded or didn't register a 
> configuration callback.
> Oct 21 10:29:53 c02 collectd[1031939]: network plugin: The 
> `MaxPacketSize' must be between 1024 and 65535.
> Oct 21 10:29:53 c02 collectd[1031939]: network plugin: Option 
> `CacheFlush' is not allowed here.
> Oct 21 10:29:53 c02 collectd[1031939]: plugin_load: plugin "interface" 

> successfully loaded.
> Oct 21 10:29:53 c02 collectd[1031939]: plugin_load: plugin "memory" 
> successfully loaded.
> Oct 21 10:29:53 c02 collectd[1031939]: plugin_load: plugin "ceph" 
> successfully loaded.
> Oct 21 10:29:53 c02 collectd[1031939]: plugin_load: plugin "ipmi" 
> successfully loaded.
> Oct 21 10:29:53 c02 collectd[1031939]: ipmi plugin: Legacy 
> configuration found! Please update your config file.
> Oct 21 10:29:53 c02 collectd[1031939]: plugin_load: plugin "exec" 
> successfully loaded.
> Oct 21 10:29:53 c02 collectd[1031939]: plugin_load: plugin "disk" 
> successfully loaded.
> Oct 21 10:29:53 c02 collectd[1031939]: Systemd detected, trying to 
> signal readyness.
> Oct 21 10:29:53 c02 systemd[1]: Started Collectd statistics daemon.
> Oct 21 10:29:53 c02 collectd[1031939]: Initialization complete, 
> entering read-loop.
> Oct 21 10:30:04 c02 collectd[1031939]: ipmi plugin: sensor_list_add: 
> Ignore sensor `PS2 Status power_supply (10.2)` of `main`, because it 
> is discrete (0x8)! Its t

Re: [ceph-users] collectd Ceph metric

[Marc Roos]

: 
sensor `P2-DIMMH3 TEMP memory_device (32.94)` of `main` not present.
Oct 21 10:30:55 c02 collectd[1031939]: ipmi plugin: sensor_read_handler: 
sensor `P2-DIMMH2 TEMP memory_device (32.93)` of `main` not present.
Oct 21 10:30:55 c02 collectd[1031939]: ipmi plugin: sensor_read_handler: 
sensor `P2-DIMMG3 TEMP memory_device (32.90)` of `main` not present.
Oct 21 10:30:55 c02 collectd[1031939]: ipmi plugin: sensor_read_handler: 
sensor `P2-DIMMG2 TEMP memory_device (32.89)` of `main` not present.
Oct 21 10:30:55 c02 collectd[1031939]: ipmi plugin: sensor_read_handler: 
sensor `P2-DIMMF3 TEMP memory_device (32.86)` of `main` not present.
Oct 21 10:30:55 c02 collectd[1031939]: ipmi plugin: sensor_read_handler: 
sensor `P2-DIMME3 TEMP memory_device (32.82)` of `main` not present.
Oct 21 10:30:55 c02 collectd[1031939]: ipmi plugin: sensor_read_handler: 
sensor `P1-DIMMD3 TEMP memory_device (32.78)` of `main` not present.
Oct 21 10:30:55 c02 collectd[1031939]: ipmi plugin: sensor_read_handler: 
sensor `P1-DIMMD2 TEMP memory_device (32.77)` of `main` not present.
Oct 21 10:30:55 c02 collectd[1031939]: ipmi plugin: sensor_read_handler: 
sensor `P1-DIMMC3 TEMP memory_device (32.74)` of `main` not present.
Oct 21 10:30:55 c02 collectd[1031939]: ipmi plugin: sensor_read_handler: 
sensor `P1-DIMMC2 TEMP memory_device (32.73)` of `main` not present.





-Original Message-
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] collectd Ceph metric

On 10:16 Mon 21 Oct, Marc Roos wrote:
> I have the same. I do not think ConvertSpecialMetricTypes is 
necessary. 
> 
> 
>   Globals true
> 
> 
> 
>   LongRunAvgLatency false
>   ConvertSpecialMetricTypes true
>   
> SocketPath "/var/run/ceph/ceph-osd.1.asok"
>   
> 
Same configuration, but there's below error after "systemctl restart 
collectd"
Have you ever hit this error before?

=Log start===
Oct 21 16:22:52 rdmarhel0 collectd[768000]: Found a configuration for 
the `ceph' plugin, but the plugin isn't loaded or didn't register a 
configuration callback.
Oct 21 16:22:52 rdmarhel0 systemd[1]: Unit collectd.service entered 
failed state.
Oct 21 16:22:52 rdmarhel0 collectd[768000]: Found a configuration for 
the `ceph' plugin, but the plugin isn't loaded or didn't register a 
configuration callback.
Oct 21 16:22:52 rdmarhel0 systemd[1]: collectd.service failed.
Oct 21 16:22:52 rdmarhel0 collectd[768000]: There is a `Daemon' block 
within the configuration for the ceph plugin. The plugin either only 
expects "simple" configuration statements or wasn Oct 21 16:22:52 
rdmarhel0 systemd[1]: collectd.service holdoff time over, scheduling 
restart.
Oct 21 16:22:52 rdmarhel0 systemd[1]: Stopped Collectd statistics 
daemon.
-- Subject: Unit collectd.service has finished shutting down 
=Log end===

B.R.
Changcheng
> 
> 
> -Original Message-
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] collectd Ceph metric
> 
> On 09:50 Mon 21 Oct, Marc Roos wrote:
> > 
> > I am, collectd with luminous, and upgraded to nautilus and collectd
> > 5.8.1-1.el7 this weekend. Maybe increase logging or so. 
> > I had to wait a long time before collectd was supporting the 
> > luminous release, maybe it is the same with octopus (=15?)
> > 
> @Roos: Do you mean that you could run collectd(5.8.1) with 
> Ceph-Nautilus? Below is my collectd configuration with Ceph-Octopus:
> 
> 
> 
>   
> SocketPath "/var/run/ceph/ceph-osd.0.asok"
>   
> 
> 
> Is there anything wrong?
> 
> > 
> >  
> > 
> > -Original Message-
> > From: Liu, Changcheng [mailto:changcheng@intel.com]
> > Sent: maandag 21 oktober 2019 9:41
> > To: ceph-users@lists.ceph.com
> > Subject: [ceph-users] collectd Ceph metric
> > 
> > Hi all,
> >Does anyone succeed to use collectd/ceph plugin to collect ceph
> >cluster data?
> >I'm using collectd(5.8.1) and Ceph-15.0.0. collectd failed to get
> >cluster data with below error:
> >"collectd.service holdoff time over, scheduling restart"
> > 
> > Regards,
> > Changcheng
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > 
> > 
> 
> 


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] collectd Ceph metric

[Marc Roos]

I have the same. I do not think ConvertSpecialMetricTypes is necessary. 


  Globals true



  LongRunAvgLatency false
  ConvertSpecialMetricTypes true
  
SocketPath "/var/run/ceph/ceph-osd.1.asok"
  



-Original Message-
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] collectd Ceph metric

On 09:50 Mon 21 Oct, Marc Roos wrote:
> 
> I am, collectd with luminous, and upgraded to nautilus and collectd
> 5.8.1-1.el7 this weekend. Maybe increase logging or so. 
> I had to wait a long time before collectd was supporting the luminous 
> release, maybe it is the same with octopus (=15?)
> 
@Roos: Do you mean that you could run collectd(5.8.1) with 
Ceph-Nautilus? Below is my collectd configuration with Ceph-Octopus:



  
SocketPath "/var/run/ceph/ceph-osd.0.asok"
  


Is there anything wrong?

> 
>  
> 
> -Original Message-
> From: Liu, Changcheng [mailto:changcheng@intel.com]
> Sent: maandag 21 oktober 2019 9:41
> To: ceph-users@lists.ceph.com
> Subject: [ceph-users] collectd Ceph metric
> 
> Hi all,
>Does anyone succeed to use collectd/ceph plugin to collect ceph
>cluster data?
>I'm using collectd(5.8.1) and Ceph-15.0.0. collectd failed to get
>cluster data with below error:
>"collectd.service holdoff time over, scheduling restart"
> 
> Regards,
> Changcheng
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] collectd Ceph metric

[Marc Roos]

I am, collectd with luminous, and upgraded to nautilus and collectd 
5.8.1-1.el7 this weekend. Maybe increase logging or so. 
I had to wait a long time before collectd was supporting the luminous 
release, maybe it is the same with octopus (=15?)


 

-Original Message-
From: Liu, Changcheng [mailto:changcheng@intel.com] 
Sent: maandag 21 oktober 2019 9:41
To: ceph-users@lists.ceph.com
Subject: [ceph-users] collectd Ceph metric

Hi all,
   Does anyone succeed to use collectd/ceph plugin to collect ceph
   cluster data?
   I'm using collectd(5.8.1) and Ceph-15.0.0. collectd failed to get
   cluster data with below error:
   "collectd.service holdoff time over, scheduling restart"

Regards,
Changcheng
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph pg repair clone_missing?

[Marc Roos]

 
Brad, many thanks!!! My cluster has finally HEALTH_OK af 1,5 year or so! 
:)


-Original Message-
Subject: Re: Ceph pg repair clone_missing?

On Fri, Oct 4, 2019 at 6:09 PM Marc Roos  
wrote:
>
>  >
>  >Try something like the following on each OSD that holds a copy of
>  >rbd_data.1f114174b0dc51.0974 and see what output you 
get.
>  >Note that you can drop the bluestore flag if they are not bluestore  

> >osds and you will need the osd stopped at the time (set noout). Also  

> >note, snapids are displayed in hexadecimal in the output (but then 
'4'
>  >is '4' so not a big issues here).
>  >
>  >$ ceph-objectstore-tool --type bluestore --data-path  
> >/var/lib/ceph/osd/ceph-XX/ --pgid 17.36 --op list
>  >rbd_data.1f114174b0dc51.0974
>
> I got these results
>
> osd.7
> Error getting attr on : 17.36_head,#-19:6c00:::scrub_17.36:head#,
> (61) No data available
> ["17.36",{"oid":"rbd_data.1f114174b0dc51.0974","key":"","s
> na 
> pid":63,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]
> ["17.36",{"oid":"rbd_data.1f114174b0dc51.0974","key":"","s
> na 
> pid":-2,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]

Ah, so of course the problem is the snapshot is missing. You may need to 
try something like the following on each of those osds.

$ ceph-objectstore-tool --type bluestore --data-path 
/var/lib/ceph/osd/ceph-XX/ --pgid 17.36 
'{"oid":"rbd_data.1f114174b0dc51.0974","key":"","snapid":-2,
"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}'
remove-clone-metadata 4

>
> osd.12
> ["17.36",{"oid":"rbd_data.1f114174b0dc51.0974","key":"","s
> na 
> pid":63,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]
> ["17.36",{"oid":"rbd_data.1f114174b0dc51.0974","key":"","s
> na 
> pid":-2,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]
>
> osd.29
> ["17.36",{"oid":"rbd_data.1f114174b0dc51.0974","key":"","s
> na 
> pid":63,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]
> ["17.36",{"oid":"rbd_data.1f114174b0dc51.0974","key":"","s
> na 
> pid":-2,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]
>
>
>  >
>  >The likely issue here is the primary believes snapshot 4 is gone but 
 
> >there is still data and/or metadata on one of the replicas which is  
> >confusing the issue. If that is the case you can use the the  
> >ceph-objectstore-tool to delete the relevant snapshot(s)  >



--
Cheers,
Brad



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph pg repair clone_missing?

[Marc Roos]

 >
 >Try something like the following on each OSD that holds a copy of
 >rbd_data.1f114174b0dc51.0974 and see what output you get.
 >Note that you can drop the bluestore flag if they are not bluestore
 >osds and you will need the osd stopped at the time (set noout). Also
 >note, snapids are displayed in hexadecimal in the output (but then '4'
 >is '4' so not a big issues here).
 >
 >$ ceph-objectstore-tool --type bluestore --data-path
 >/var/lib/ceph/osd/ceph-XX/ --pgid 17.36 --op list
 >rbd_data.1f114174b0dc51.0974

I got these results

osd.7
Error getting attr on : 17.36_head,#-19:6c00:::scrub_17.36:head#, 
(61) No data available
["17.36",{"oid":"rbd_data.1f114174b0dc51.0974","key":"","sna
pid":63,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]
["17.36",{"oid":"rbd_data.1f114174b0dc51.0974","key":"","sna
pid":-2,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]

osd.12
["17.36",{"oid":"rbd_data.1f114174b0dc51.0974","key":"","sna
pid":63,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]
["17.36",{"oid":"rbd_data.1f114174b0dc51.0974","key":"","sna
pid":-2,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]

osd.29
["17.36",{"oid":"rbd_data.1f114174b0dc51.0974","key":"","sna
pid":63,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]
["17.36",{"oid":"rbd_data.1f114174b0dc51.0974","key":"","sna
pid":-2,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]


 >
 >The likely issue here is the primary believes snapshot 4 is gone but
 >there is still data and/or metadata on one of the replicas which is
 >confusing the issue. If that is the case you can use the the
 >ceph-objectstore-tool to delete the relevant snapshot(s)
 >
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] NFS

[Marc Roos]

Thanks Matt! Really useful configs. I am still on luminous, so I can 
forget about this now :( I will try when I am nautilus, I have already 
updated my configuration. However it is interesting that in the 
configuration nowhere the tenant is specified, so I guess that is being 
extracted from the access key/irrelevant.



-Original Message-
Subject: Re: [ceph-users] NFS

Hi Mark,

Here's an example that should work--userx and usery are RGW users 
created in different tenants, like so:

radosgw-admin --tenant tnt1 --uid userx --display-name "tnt1-userx" \
 --access_key "userxacc" --secret "test123" user create

 radosgw-admin --tenant tnt2 --uid usery --display-name "tnt2-usery" \
 --access_key "useryacc" --secret "test456" user create

Remember that to make use of this feature, you need recent librgw and 
matching nfs-ganesha.  In particular, Ceph should have, among other
changes:

commit 65d0ae733defe277f31825364ee52d5102c06ab9
Author: Matt Benjamin 
Date:   Wed Jun 5 07:25:35 2019 -0400

rgw_file: include tenant in hashes of object

Because bucket names are taken as object names in the top
of an export.  Make hashing by tenant general to avoid disjoint
hashing of bucket.

Fixes: http://tracker.ceph.com/issues/40118

Signed-off-by: Matt Benjamin 
(cherry picked from commit 8e0fd5fbfa7c770f6b668e79b772179946027bce)

commit 459b6b2b224953655fd0360e8098ae598e41d3b2
Author: Matt Benjamin 
Date:   Wed May 15 15:53:32 2019 -0400

rgw_file: include tenant when hashing bucket names

Prevent identical paths from distinct tenants from colliding in
RGW NFS handle cache.

Fixes: http://tracker.ceph.com/issues/40118

Signed-off-by: Matt Benjamin 
(cherry picked from commit b800a9de83dff23a150ed7d236cb61c8b7d971ae)
Signed-off-by: Matt Benjamin 


ganesha.conf.deuxtenant:


EXPORT
{
# Export Id (mandatory, each EXPORT must have a unique Export_Id)
Export_Id = 77;

# Exported path (mandatory)
Path = "/";

# Pseudo Path (required for NFS v4)
Pseudo = "/userx";

# Required for access (default is None)
# Could use CLIENT blocks instead
Access_Type = RW;

SecType = "sys";

Protocols = 3,4;
Transports = UDP,TCP;

#Delegations = Readwrite;

Squash = No_Root_Squash;

# Exporting FSAL
FSAL {
Name = RGW;
User_Id = "userx";
Access_Key_Id = "userxacc";
Secret_Access_Key = "test123";
}
}

EXPORT
{
# Export Id (mandatory, each EXPORT must have a unique Export_Id)
Export_Id = 78;

# Exported path (mandatory)
Path = "/";

# Pseudo Path (required for NFS v4)
Pseudo = "/usery";

# Required for access (default is None)
# Could use CLIENT blocks instead
Access_Type = RW;

SecType = "sys";

Protocols = 3,4;
Transports = UDP,TCP;

#Delegations = Readwrite;

Squash = No_Root_Squash;

# Exporting FSAL
FSAL {
Name = RGW;
User_Id = "usery";
Access_Key_Id = "useryacc";
Secret_Access_Key = "test456";
}
}

#mount at bucket case
EXPORT
{
# Export Id (mandatory, each EXPORT must have a unique Export_Id)
Export_Id = 79;

# Exported path (mandatory)
Path = "/buck5";

# Pseudo Path (required for NFS v4)
Pseudo = "/usery_buck5";

# Required for access (default is None)
# Could use CLIENT blocks instead
Access_Type = RW;

SecType = "sys";

Protocols = 3,4;
Transports = UDP,TCP;

#Delegations = Readwrite;

Squash = No_Root_Squash;

# Exporting FSAL
FSAL {
Name = RGW;
User_Id = "usery";
Access_Key_Id = "useryacc";
Secret_Access_Key = "test456";
}
}



RGW {
ceph_conf = "/home/mbenjamin/ceph-noob/build/ceph.conf";
#init_args = "-d --debug-rgw=16";
init_args = "";
}

NFS_Core_Param {
Nb_Worker = 17;
mount_path_pseudo = true;
}

CacheInode {
Chunks_HWMark = 7;
Entries_Hwmark = 200;
}

NFSV4 {
Graceless = true;
Allow_Numeric_Owners = true;
Only_Numeric_Owners = true;
}

LOG {
Components {
#NFS_READDIR = FULL_DEBUG;
#NFS4 = FULL_DEBUG;
#CACHE_INODE = FULL_DEBUG;
#FSAL = FULL_DEBUG;
}
Facility {
name = FILE;
destination = "/tmp/ganesha-rgw.log";
enable = active;
}
}

On Thu, Oct 3, 2019 at 10:34 AM Marc Roos  
wrote:
>
>
> How should a multi tenant RGW config look like, I am not able get this
> working:
>
> EXPORT {
>Export_ID=301;
>Path = "test:test3";
>#Path = "/";
>Pseudo = "/rgwtester";
>
>   

Re: [ceph-users] NFS

[Marc Roos]

How should a multi tenant RGW config look like, I am not able get this 
working:

EXPORT {
   Export_ID=301;
   Path = "test:test3";
   #Path = "/";
   Pseudo = "/rgwtester";

   Protocols = 4;
   FSAL {
   Name = RGW;
   User_Id = "test$tester1";
   Access_Key_Id = "TESTER";
   Secret_Access_Key = "xxx";
   }
   Disable_ACL = TRUE;
   CLIENT { Clients = 192.168.10.0/24; access_type = "RO"; }
}


03/10/2019 16:15:37 : epoch 5d8d274c : c01 : ganesha.nfsd-4722[sigmgr] 
create_export :FSAL :CRIT :RGW module: librgw init failed (-5)
03/10/2019 16:15:37 : epoch 5d8d274c : c01 : ganesha.nfsd-4722[sigmgr] 
mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on 
underlying FSAL RGW
03/10/2019 16:15:37 : epoch 5d8d274c : c01 : ganesha.nfsd-4722[sigmgr] 
fsal_put :FSAL :INFO :FSAL RGW now unused
03/10/2019 16:15:37 : epoch 5d8d274c : c01 : ganesha.nfsd-4722[sigmgr] 
fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/rgwtester) 
to (test:test3)
03/10/2019 16:15:37 : epoch 5d8d274c : c01 : ganesha.nfsd-4722[sigmgr] 
fsal_cfg_commit :FSAL :F_DBG :FSAL RGW refcount 0
03/10/2019 16:15:37 : epoch 5d8d274c : c01 : ganesha.nfsd-4722[sigmgr] 
config_errs_to_log :CONFIG :CRIT :Config File 
(/etc/ganesha/ganesha.conf:216): 1 validation errors in block FSAL
03/10/2019 16:15:37 : epoch 5d8d274c : c01 : ganesha.nfsd-4722[sigmgr] 
config_errs_to_log :CONFIG :CRIT :Config File 
(/etc/ganesha/ganesha.conf:216): Errors processing block (FSAL)
03/10/2019 16:15:37 : epoch 5d8d274c : c01 : ganesha.nfsd-4722[sigmgr] 
config_errs_to_log :CONFIG :CRIT :Config File 
(/etc/ganesha/ganesha.conf:209): 1 validation errors in block EXPORT
03/10/2019 16:15:37 : epoch 5d8d274c : c01 : ganesha.nfsd-4722[sigmgr] 
config_errs_to_log :CONFIG :CRIT :Config File 
(/etc/ganesha/ganesha.conf:209): Errors processing block (EXPORT)

-Original Message-
Subject: Re: [ceph-users] NFS

RGW NFS can support any NFS style of authentication, but users will have 
the RGW access of their nfs-ganesha export.  You can create exports with 
disjoint privileges, and since recent L, N, RGW tenants.

Matt

On Tue, Oct 1, 2019 at 8:31 AM Marc Roos  
wrote:
>
>  I think you can run into problems
> with a multi user environment of RGW and nfs-ganesha.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph pg repair clone_missing?

[Marc Roos]

 >
 >>
 >> I was following the thread where you adviced on this pg repair
 >>
 >> I ran these rados 'list-inconsistent-obj'/'rados 
 >> list-inconsistent-snapset' and have output on the snapset. I tried 
to 
 >> extrapolate your comment on the data/omap_digest_mismatch_info onto 
my 
 >> situation. But I don't know how to proceed. I got on this mailing 
list 
 >> the advice to delete snapshot 4, but if I see this output, that 
might 
 >> not have been the smartest thing to do.
 >
 >That remains to be seen. Can you post the actual scrub error you are 
getting?

2019-10-03 09:27:07.831046 7fc448bf6700 -1 log_channel(cluster) log 
[ERR] : deep-scrub 17.36 
17:6ca1f70a:::rbd_data.1f114174b0dc51.0974:head : expected 
clone 17:6ca1f70a:::rbd_data.1f114174b0dc51.0974:4 1 missing

 >>
 >>
 >>
 >>
 >> [0]
 >> http://tracker.ceph.com/issues/24994
 >
 >At first glance this appears to be a different issue to yours.
 >
 >>
 >> [1]
 >> {
 >>   "epoch": 66082,
 >>   "inconsistents": [
 >> {
 >>   "name": "rbd_data.1f114174b0dc51.0974",
 >
 >rbd_data.1f114174b0dc51 is the block_name_prefix for this image. You 
 >can run 'rbd info' on the images in this pool to see which image is
 >actually affected and how important the data is.

Yes I know what image it is. Deleting data is easy, I like to know/learn 

how to fix this.

 >
 >>   "nspace": "",
 >>   "locator": "",
 >>   "snap": "head",
 >>   "snapset": {
 >> "snap_context": {
 >>   "seq": 63,
 >>   "snaps": [
 >> 63,
 >> 35,
 >> 13,
 >> 4
 >>   ]
 >> },
 >> "head_exists": 1,
 >> "clones": [
 >>   {
 >> "snap": 4,
 >> "size": 4194304,
 >> "overlap": "[]",
 >> "snaps": [
 >>   4
 >> ]
 >>   },
 >>   {
 >> "snap": 63,
 >> "size": 4194304,
 >> "overlap": "[0~4194304]",
 >> "snaps": [
 >>   63,
 >>   35,
 >>   13
 >> ]
 >>   }
 >> ]
 >>   },
 >>   "errors": [
 >> "clone_missing"
 >>   ],
 >>   "missing": [
 >> 4
 >>   ]
 >> }
 >>   ]
 >> }
 >
 >
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph pg repair clone_missing?

[Marc Roos]

 
Hi Brad, 

I was following the thread where you adviced on this pg repair

I ran these rados 'list-inconsistent-obj'/'rados 
list-inconsistent-snapset' and have output on the snapset. I tried to 
extrapolate your comment on the data/omap_digest_mismatch_info onto my 
situation. But I don't know how to proceed. I got on this mailing list 
the advice to delete snapshot 4, but if I see this output, that might 
not have been the smartest thing to do.




[0]
http://tracker.ceph.com/issues/24994

[1]
{
  "epoch": 66082,
  "inconsistents": [
{
  "name": "rbd_data.1f114174b0dc51.0974",
  "nspace": "",
  "locator": "",
  "snap": "head",
  "snapset": {
"snap_context": {
  "seq": 63,
  "snaps": [
63,
35,
13,
4
  ]
},
"head_exists": 1,
"clones": [
  {
"snap": 4,
"size": 4194304,
"overlap": "[]",
"snaps": [
  4
]
  },
  {
"snap": 63,
"size": 4194304,
"overlap": "[0~4194304]",
"snaps": [
  63,
  35,
  13
]
  }
]
  },
  "errors": [
"clone_missing"
  ],
  "missing": [
4
  ]
}
  ]
}
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] NFS

[Marc Roos]

Yes indeed cephfs and rgw backends. I think you can run into problems 
with a multi user environment of RGW and nfs-ganesha. I am not getting 
this working on Luminous. Your rgw config seems ok. 

Add file logging to debug rgw etc something like this.


LOG {
## Default log level for all components
# NULL, FATAL, MAJ, CRIT, WARN, EVENT, INFO, DEBUG, MID_DEBUG, 
M_DBG, FULL_DEBUG, F_DBG], default EVENT
default_log_level = INFO;
#default_log_level = DEBUG;

## Configure per-component log levels.
# ALL, 
LOG,LOG_EMERG,MEMLEAKS,FSAL,NFSPROTO(NFS3),NFS_V4(NFSV4),EXPORT,FILEHAND
LE,DISPATCH,CACHE_INODE,
# CACHE_INODE_LRU,HASHTABLE,HASHTABLE_CACHE,DUPREQ,INIT,MAIN, 
IDMAPPER,NFS_READDIR,NFS_V4_LOCK,CONFIG,CLIENTID,
# 
SESSIONS,PNFS,RW_LOCK,NLM,RPC,NFS_CB,THREAD,NFS_V4_ACL,STATE,9P,9P_DISPA
TCH,FSAL_UP,DBUS

Components {
ALL = WARN;
#ALL = DEBUG;
#FSAL = F_DBG;
#NFS4 = F_DBG;
#EXPORT = F_DBG;
#CONFIG = F_DBG;
}

## Where to log
#   Facility {
#   name = FILE;
#   destination = "/var/log/ganesha.log";
#   enable = default;
#   }
}

-Original Message-
Subject: Re: [ceph-users] NFS

Ganesha can export CephFS or RGW.  It cannot export anything else (like 
iscsi or RBD).  Config for RGW looks like this:

EXPORT
{
 Export_ID=1;
 Path = "/";
 Pseudo = "/rgw";
 Access_Type = RW;
 Protocols = 4;
 Transports = TCP;
 FSAL {
 Name = RGW;
 User_Id = "testuser";
 Access_Key_Id ="";
 Secret_Access_Key = "";
 }
}

RGW {
 ceph_conf = "//ceph.conf";
 # for vstart cluster, name = "client.admin"
 name = "client.rgw.foohost";
 cluster = "ceph";
#   init_args = "-d --debug-rgw=16";
}


Daniel

On 9/30/19 3:01 PM, Marc Roos wrote:
>   
> Just install these
> 
> http://download.ceph.com/nfs-ganesha/
> nfs-ganesha-rgw-2.7.1-0.1.el7.x86_64
> nfs-ganesha-vfs-2.7.1-0.1.el7.x86_64
> libnfsidmap-0.25-19.el7.x86_64
> nfs-ganesha-mem-2.7.1-0.1.el7.x86_64
> nfs-ganesha-xfs-2.7.1-0.1.el7.x86_64
> nfs-ganesha-2.7.1-0.1.el7.x86_64
> nfs-ganesha-ceph-2.7.1-0.1.el7.x86_64
> 
> 
> And export your cephfs like this:
> EXPORT {
>  Export_Id = 10;
>  Path = /nfs/cblr-repos;
>  Pseudo = /cblr-repos;
>  FSAL { Name = CEPH; User_Id = "cephfs.nfs.cblr"; 
> Secret_Access_Key = "xxx"; }
>  Disable_ACL = FALSE;
>  CLIENT { Clients = 192.168.10.2; access_type = "RW"; }
>  CLIENT { Clients = 192.168.10.253; } }
> 
> 
> -Original Message-
> From: Brent Kennedy [mailto:bkenn...@cfl.rr.com]
> Sent: maandag 30 september 2019 20:56
> To: 'ceph-users'
> Subject: [ceph-users] NFS
> 
> Wondering if there are any documents for standing up NFS with an 
> existing ceph cluster.  We don’t use ceph-ansible or any other tools 
> besides ceph-deploy.  The iscsi directions were pretty good once I got 

> past the dependencies.
> 
>   
> 
> I saw the one based on Rook, but it doesn’t seem to apply to our 
setup 
> of ceph vms with physical hosts doing OSDs.  The official ceph 
> documents talk about using ganesha but doesn’t seem to dive into the 
> details of what the process is for getting it online.  We don’t use 
> cephfs, so that’s not setup either.  The basic docs seem to note this 
is required.
>   Seems my google-fu is failing me when I try to find a more 
> definitive guide.
> 
>   
> 
> The servers are all centos 7 with the latest updates.
> 
>   
> 
> Any guidance would be greatly appreciated!
> 
>   
> 
> Regards,
> 
> -Brent
> 
>   
> 
> Existing Clusters:
> 
> Test: Nautilus 14.2.2 with 3 osd servers, 1 mon/man, 1 gateway, 2 
> iscsi gateways ( all virtual on nvme )
> 
> US Production(HDD): Nautilus 14.2.2 with 13 osd servers, 3 mons, 4 
> gateways, 2 iscsi gateways
> 
> UK Production(HDD): Nautilus 14.2.2 with 25 osd servers, 3 mons/man, 3 

> gateways behind
> 
> US Production(SSD): Nautilus 14.2.2 with 6 osd servers, 3 mons/man, 3 
> gateways, 2 iscsi gateways
> 
>   
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] NFS

[Marc Roos]

 
Just install these

http://download.ceph.com/nfs-ganesha/
nfs-ganesha-rgw-2.7.1-0.1.el7.x86_64
nfs-ganesha-vfs-2.7.1-0.1.el7.x86_64
libnfsidmap-0.25-19.el7.x86_64
nfs-ganesha-mem-2.7.1-0.1.el7.x86_64
nfs-ganesha-xfs-2.7.1-0.1.el7.x86_64
nfs-ganesha-2.7.1-0.1.el7.x86_64
nfs-ganesha-ceph-2.7.1-0.1.el7.x86_64


And export your cephfs like this:
EXPORT {
Export_Id = 10;
Path = /nfs/cblr-repos;
Pseudo = /cblr-repos;
FSAL { Name = CEPH; User_Id = "cephfs.nfs.cblr"; 
Secret_Access_Key = "xxx"; }
Disable_ACL = FALSE;
CLIENT { Clients = 192.168.10.2; access_type = "RW"; }
CLIENT { Clients = 192.168.10.253; }
}


-Original Message-
From: Brent Kennedy [mailto:bkenn...@cfl.rr.com] 
Sent: maandag 30 september 2019 20:56
To: 'ceph-users'
Subject: [ceph-users] NFS

Wondering if there are any documents for standing up NFS with an 
existing ceph cluster.  We don’t use ceph-ansible or any other tools 
besides ceph-deploy.  The iscsi directions were pretty good once I got 
past the dependencies.  

 

I saw the one based on Rook, but it doesn’t seem to apply to our setup 
of ceph vms with physical hosts doing OSDs.  The official ceph documents 
talk about using ganesha but doesn’t seem to dive into the details of 
what the process is for getting it online.  We don’t use cephfs, so 
that’s not setup either.  The basic docs seem to note this is required. 
 Seems my google-fu is failing me when I try to find a more definitive 
guide.

 

The servers are all centos 7 with the latest updates.

 

Any guidance would be greatly appreciated!

 

Regards,

-Brent

 

Existing Clusters:

Test: Nautilus 14.2.2 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi 
gateways ( all virtual on nvme )

US Production(HDD): Nautilus 14.2.2 with 13 osd servers, 3 mons, 4 
gateways, 2 iscsi gateways

UK Production(HDD): Nautilus 14.2.2 with 25 osd servers, 3 mons/man, 3 
gateways behind

US Production(SSD): Nautilus 14.2.2 with 6 osd servers, 3 mons/man, 3 
gateways, 2 iscsi gateways

 


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Commit and Apply latency on nautilus

[Marc Roos]

What parameters are you exactly using? I want to do a similar test on 
luminous, before I upgrade to Nautilus. I have quite a lot (74+)

type_instance=Osd.opBeforeDequeueOpLat
type_instance=Osd.opBeforeQueueOpLat
type_instance=Osd.opLatency
type_instance=Osd.opPrepareLatency
type_instance=Osd.opProcessLatency
type_instance=Osd.opRLatency
type_instance=Osd.opRPrepareLatency
type_instance=Osd.opRProcessLatency
type_instance=Osd.opRwLatency
type_instance=Osd.opRwPrepareLatency
type_instance=Osd.opRwProcessLatency
type_instance=Osd.opWLatency
type_instance=Osd.opWPrepareLatency
type_instance=Osd.opWProcessLatency
type_instance=Osd.subopLatency
type_instance=Osd.subopWLatency
...
...





-Original Message-
From: Alex Litvak [mailto:alexander.v.lit...@gmail.com] 
Sent: zondag 29 september 2019 13:06
To: ceph-users@lists.ceph.com
Cc: ceph-de...@vger.kernel.org
Subject: [ceph-users] Commit and Apply latency on nautilus

Hello everyone,

I am running a number of parallel benchmark tests against the cluster 
that should be ready to go to production.
I enabled prometheus to monitor various information and while cluster 
stays healthy through the tests with no errors or slow requests,
I noticed an apply / commit latency jumping between 40 - 600 ms on 
multiple SSDs.  At the same time op_read and op_write are on average 
below 0.25 ms in the worth case scenario.

I am running nautilus 14.2.2, all bluestore, no separate NVME devices 
for WAL/DB, 6 SSDs per node(Dell PowerEdge R440) with all drives Seagate 
Nytro 1551, osd spread across 6 nodes, running in 
containers.  Each node has plenty of RAM with utilization ~ 25 GB during 
the benchmark runs.

Here are benchmarks being run from 6 client systems in parallel, 
repeating the test for each block size in <4k,16k,128k,4M>.

On rbd mapped partition local to each client:

fio --name=randrw --ioengine=libaio --iodepth=4 --rw=randrw 
--bs=<4k,16k,128k,4M> --direct=1 --size=2G --numjobs=8 --runtime=300 
--group_reporting --time_based --rwmixread=70

On mounted cephfs volume with each client storing test file(s) in own 
sub-directory:

fio --name=randrw --ioengine=libaio --iodepth=4 --rw=randrw 
--bs=<4k,16k,128k,4M> --direct=1 --size=2G --numjobs=8 --runtime=300 
--group_reporting --time_based --rwmixread=70

dbench -t 30 30

Could you please let me know if huge jump in applied and committed 
latency is justified in my case and whether I can do anything to improve 
/ fix it.  Below is some additional cluster info.

Thank you,

root@storage2n2-la:~# podman exec -it ceph-mon-storage2n2-la ceph osd df
ID CLASS WEIGHT  REWEIGHT SIZERAW USE DATAOMAPMETA AVAIL 
  %USE VAR  PGS STATUS
  6   ssd 1.74609  1.0 1.7 TiB  93 GiB  92 GiB 240 MiB  784 MiB 1.7 
TiB 5.21 0.90  44 up
12   ssd 1.74609  1.0 1.7 TiB  98 GiB  97 GiB 118 MiB  906 MiB 1.7 
TiB 5.47 0.95  40 up
18   ssd 1.74609  1.0 1.7 TiB 102 GiB 101 GiB 123 MiB  901 MiB 1.6 
TiB 5.73 0.99  47 up
24   ssd 3.49219  1.0 3.5 TiB 222 GiB 221 GiB 134 MiB  890 MiB 3.3 
TiB 6.20 1.07  96 up
30   ssd 3.49219  1.0 3.5 TiB 213 GiB 212 GiB 151 MiB  873 MiB 3.3 
TiB 5.95 1.03  93 up
35   ssd 3.49219  1.0 3.5 TiB 203 GiB 202 GiB 301 MiB  723 MiB 3.3 
TiB 5.67 0.98 100 up
  5   ssd 1.74609  1.0 1.7 TiB 103 GiB 102 GiB 123 MiB  901 MiB 1.6 
TiB 5.78 1.00  49 up
11   ssd 1.74609  1.0 1.7 TiB 109 GiB 108 GiB  63 MiB  961 MiB 1.6 
TiB 6.09 1.05  46 up
17   ssd 1.74609  1.0 1.7 TiB 104 GiB 103 GiB 205 MiB  819 MiB 1.6 
TiB 5.81 1.01  50 up
23   ssd 3.49219  1.0 3.5 TiB 210 GiB 209 GiB 168 MiB  856 MiB 3.3 
TiB 5.86 1.01  86 up
29   ssd 3.49219  1.0 3.5 TiB 204 GiB 203 GiB 272 MiB  752 MiB 3.3 
TiB 5.69 0.98  92 up
34   ssd 3.49219  1.0 3.5 TiB 198 GiB 197 GiB 295 MiB  729 MiB 3.3 
TiB 5.54 0.96  85 up
  4   ssd 1.74609  1.0 1.7 TiB 119 GiB 118 GiB  16 KiB 1024 MiB 1.6 
TiB 6.67 1.15  50 up
10   ssd 1.74609  1.0 1.7 TiB  95 GiB  94 GiB 183 MiB  841 MiB 1.7 
TiB 5.31 0.92  46 up
16   ssd 1.74609  1.0 1.7 TiB 102 GiB 101 GiB 122 MiB  902 MiB 1.6 
TiB 5.72 0.99  50 up
22   ssd 3.49219  1.0 3.5 TiB 218 GiB 217 GiB 109 MiB  915 MiB 3.3 
TiB 6.11 1.06  91 up
28   ssd 3.49219  1.0 3.5 TiB 198 GiB 197 GiB 343 MiB  681 MiB 3.3 
TiB 5.54 0.96  95 up
33   ssd 3.49219  1.0 3.5 TiB 198 GiB 196 GiB 297 MiB 1019 MiB 3.3 
TiB 5.53 0.96  85 up
  1   ssd 1.74609  1.0 1.7 TiB 101 GiB 100 GiB 222 MiB  802 MiB 1.6 
TiB 5.63 0.97  49 up
  7   ssd 1.74609  1.0 1.7 TiB 102 GiB 101 GiB 153 MiB  871 MiB 1.6 
TiB 5.69 0.99  46 up
13   ssd 1.74609  1.0 1.7 TiB 106 GiB 105 GiB  67 MiB  957 MiB 1.6 
TiB 5.96 1.03  42 up
19   ssd 3.49219  1.0 3.5 TiB 206 GiB 205 GiB 179 MiB  845 MiB 3.3 
TiB 5.77 1.00  83 up
25   ssd 3.49219  1.0 3.5 TiB 195 GiB 194 GiB 352 MiB  672 MiB 3.3 
TiB 5.45 0.94  97 up
31   ssd 3.49219  1.0 3.5 TiB 201 GiB 200 GiB 305 MiB  719 MiB 3.3 
TiB 

Re: [ceph-users] Need advice with setup planning

[Marc Roos]

 >
 >> > -   Use 2 HDDs for SO using RAID 1 (I've left 3.5TB unallocated in 
case
 >>
 >> I can use it later for storage)
 >>
 >> OS not? get enterprise ssd as os (I think some recommend it when
 >> colocating monitors, can generate a lot of disk io)
 >
 >Yes, OS. I have no option to get a SSD.
 
one samsung ssd sm863 of 240GB on ebay is 180us$. How much are your 2x 
hdd

 >
 >>
 >> > -   Install CentOS 7.7
 >>
 >> Good choice
 >>
 >> > -   Use 2 vLANs, one for ceph internal usage and another for 
external
 >>
 >> access. Since they've 4 network adapters, I'll try to bond them in 
pairs
 >> to speed up network (1Gb).
 >>
 >> Bad, get 10Gbit, yes really
 >
 >Again, that's not an option. We'll have to use the hardware we got.

Maybe you can try and convince ceph development to optimize for bonding 
on 1gbit
beware of this:  
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg35474.html
Make sure you test your requirements, because ceph is quite some 
overhead.

 >
 >>
 >> > -   I'll try to use ceph-ansible for installation. I failed to use 
it on
 >>
 >> lab, but it seems more recommended.
 >>
 >> Where did you get it from that ansible is recommended? Ansible is a 
tool
 >> to help you automate deployments, but I have the impression it is 
mostly
 >> used as a 'I do not know how to install something' so lets use 
ansible
 >> tool.
 >
 >From reaing various sites/guides for lab.
 >
 >>
 >> > -   Install Ceph Nautilus
 >>
 >> >
 >>
 >> > -   Each server will host OSD, MON, MGR and MDS.
 >>
 >> > -   One VM for ceph-admin: This wil be used to run ceph-ansible 
and
 >>
 >> maybe to host some ceph services later
 >>
 >> Don't waste a vm on this?
 >
 >You think it is a waste to have a VM for this? Won't I need another 
machine to host other ceph services?

I am not having a vm for ceph admin. Depends on what you are going to 
do, and eg. how much
 memory you have / are using. The thing to beware of is that you could 
get kernel deadlocks 
when running tasks on osd nodes. This is being prevented by using a vm. 
However this all depends on the availability of memory. I didn't 
encounter this 
and others are running also successfully afaik.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Need advice with setup planning

[Marc Roos]

 


 >- Use 2 HDDs for SO using RAID 1 (I've left 3.5TB unallocated in case 
I can use it later for storage)

OS not? get enterprise ssd as os (I think some recommend it when 
colocating monitors, can generate a lot of disk io)

 >- Install CentOS 7.7

Good choice

 >- Use 2 vLANs, one for ceph internal usage and another for external 
access. Since they've 4 network adapters, I'll try to bond them in pairs 
to speed up network (1Gb).

Bad, get 10Gbit, yes really 

 >- I'll try to use ceph-ansible for installation. I failed to use it on 
lab, but it seems more recommended.

Where did you get it from that ansible is recommended? Ansible is a tool 
to help you automate deployments, but I have the impression it is mostly 
used as a 'I do not know how to install something' so lets use ansible 
tool.

 >- Install Ceph Nautilus
 >
 >- Each server will host OSD, MON, MGR and MDS.
 >- One VM for ceph-admin: This wil be used to run ceph-ansible and 
maybe to host some ceph services later

Don't waste a vm on this?

 >- I'll have to serve samba, iscsi and probably NFS too. Not sure how 
or on which servers.
 >

If you want to create a fancy solution, you can use something like mesos 
that manages your nfs,smb,iscsi or rgw daemons, so if you bring down a 
host, applications automatically move to a different host ;)

 





-Original Message-
From: Salsa [mailto:sa...@protonmail.com] 
Sent: vrijdag 20 september 2019 18:14
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Need advice with setup planning

I have tested Ceph using VMs but never got to put it to use and had a 
lot of trouble to get it to install.


Now I've been asked to do a production setup using 3 servers (Dell R740) 
with 12 4TB each.


My plan is this:

- Use 2 HDDs for SO using RAID 1 (I've left 3.5TB unallocated in case I 
can use it later for storage)

- Install CentOS 7.7

- Use 2 vLANs, one for ceph internal usage and another for external 
access. Since they've 4 network adapters, I'll try to bond them in pairs 
to speed up network (1Gb).

- I'll try to use ceph-ansible for installation. I failed to use it on 
lab, but it seems more recommended.

- Install Ceph Nautilus

- Each server will host OSD, MON, MGR and MDS.
- One VM for ceph-admin: This wil be used to run ceph-ansible and maybe 
to host some ceph services later
- I'll have to serve samba, iscsi and probably NFS too. Not sure how or 
on which servers.


Am I missing anything? Am I doing anything "wrong"?


I searched for some actual guidance on setup but I couldn't find 
anything complete, like a good tutorial or reference based on possible 
use-cases.


So, is there any suggestions you could share or links and references I 
should take a look?

Thanks;


--

Salsa


Sent with ProtonMail   Secure Email.




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph dovecot again

[Marc Roos]

 

How do I actually configure dovecot to use ceph for a mailbox? I have 
build the plugins as mentioned here[0] 

- but where do I copy/load what module?
- can I configure a specific mailbox only, via eg userdb:
  test3:x:8267:231:Account with special settings for 
dovecot:/home/popusers/test3:/bin/false:userdb_mail=mdbox:~/mdbox:INBOX=
/var/spool/mail/%u:INDEX=/var/dovecot/%u/index




[@test2 src]$ ls -lart librmb/.libs/

drwxrwxr-x 5 test test   4096 Sep 12 23:24 ..
-rw-rw-r-- 1 test test920 Sep 12 23:24 librmb.lai
lrwxrwxrwx 1 test test 12 Sep 12 23:24 librmb.la -> ../librmb.la
drwxrwxr-x 2 test test   4096 Sep 12 23:24 .

[@test2 src]$ ls -lart storage-rbox/.libs/
total 2076

-rwxrwxr-x 1 test test 387840 Sep 12 23:24 libstorage_rbox_plugin.so
drwxrwxr-x 4 test test   4096 Sep 12 23:24 ..
-rw-rw-r-- 1 test test   1071 Sep 12 23:24 libstorage_rbox_plugin.lai
lrwxrwxrwx 1 test test 28 Sep 12 23:24 libstorage_rbox_plugin.la -> 
../libstorage_rbox_plugin.la
drwxrwxr-x 2 test test   4096 Sep 12 23:24 .

[0]
https://github.com/ceph-dovecot/dovecot-ceph-plugin


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph performance paper

[Marc Roos]

 
Hi Vitaliy, just saw you recommend someone to use ssd, and wanted to use 
the oppurtunaty to thank you for composing this text[0], enoyed reading 
it. 

- What do you mean with: bad-SSD-only?
- Is this patch[1] in a Nautilus release?


[0]
https://yourcmc.ru/wiki/Ceph_performance

[1]
https://github.com/ceph/ceph/pull/26909
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Questions regarding backing up Ceph

[Marc Roos]

 

>
> complete DR with Ceph to restore it back to how it was at a given 
point in time is a challenge.

>
> Trying to backup a Ceph cluster sounds very 'enterprise' and is 
difficult to scale as well.

Hmmm, I was actually also curious how backups were done, especially on 
these clusters that have 300, 600 or even more osd's. 

Is it common to do backups 'within' the ceph cluster, eg with snapshots? 
Different pool? If this is common with most ceph clusters, would that 
not also increase security requirements for the development? 

Recently I read a cloud hosting provider Insynq lost all their data[0] 
One remote exploit in ceph, and TB/PB of data could be at risk. 



[0]
https://www.insynq.com/support/#status


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Future of Filestore?

[Marc Roos]

 >> Reverting back to filestore is quite a lot of work and time again. 
 >> Maybe see first if with some tuning of the vms you can get better 
results?
 >
 >None of the VMs are particularly disk-intensive.  There's two users 
accessing the system over a WiFi network for email, and some HTTP/SMTP 
traffic coming in via an ADSL2 Internet connection.
 >
 >If Bluestore can't manage this, then I'd consider it totally worthless 
in any enterprise installation -- so clearly something is wrong.


I have a cluster mainly intended for backups to cephfs, 4 nodes, sata 
disks and mostly 5400rpm. Because the cluster is doing nothing. I 
decided to put vm's on them. I am running 15 vm's without problems on 
the hdd pool. Going to move more to them. One of them is an macos 
machine, I did once a fio test in it and gave me 917 iops at 4k random 
reads. (technically not possible I would say, I have mostly default 
configurations in libvirt)


 >
 >> What you also can try is for io intensive vm's add an ssd pool?
 >
 >How well does that work in a cluster with 0 SSD-based OSDs?
 >
 >For 3 of the nodes, the cases I'm using for the servers can fit two 
2.5"
 >drives.  I have one 120GB SSD for the OS, that leaves one space spare 
for the OSD.  


I think this could be your bottle neck, I have 31 drives, so the load is 
spread across 31 (hopefully). If you have only 3 drives you have 
3x60iops to share amongst your vms. 
I am getting the impression that ceph development is not really 
interested in setups quite different from the advised standards. I once 
made an attempt to get things better working for 1Gb adapters[0].

 >
 >I since added two new nodes, which are Intel NUCs with m.2 SATA SSDs 
for the OS and like the other nodes have a single 2.5" drive bay.
 >
 >This is being done as a hobby and a learning exercise I might add -- 
so while I have spent a lot of money on this, the funds I have to throw 
at this are not infinite.


Same here ;) 


 >
 >> I moved
 >> some exchange servers on them. Tuned down the logging, because that 
is 
 >> writing constantly to disk.
 >> With such setup you are at least secured for the future.
 >
 >The VMs I have are mostly Linux (Gentoo, some Debian/Ubuntu), with a 
few OpenBSD VMs for things like routers between virtual networks.
 >

[0] https://www.mail-archive.com/ceph-users@lists.ceph.com/msg35474.html
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Future of Filestore?

[Marc Roos]

 

Reverting back to filestore is quite a lot of work and time again. Maybe 
see first if with some tuning of the vms you can get better results? 
What you also can try is for io intensive vm's add an ssd pool? I moved 
some exchange servers on them. Tuned down the logging, because that is 
writing constantly to disk. 
With such setup you are at least secured for the future.






-Original Message-
From: Stuart Longland [mailto:stua...@longlandclan.id.au] 
Subject: Re: [ceph-users] Future of Filestore?


>  
> Maybe a bit of topic, just curious what speeds did you get previously? 

> Depending on how you test your native drive of 5400rpm, the 
> performance could be similar. 4k random read of my 7200rpm/5400 rpm 
> results in ~60iops at 260kB/s.

Well, to be honest I never formally tested the performance prior to the 
move to Bluestore.  It was working "acceptably" for my needs, thus I 
never had a reason to test it.

It was never a speed demon, but it did well enough for my needs.  Had 
Filestore on BTRFS remained an option in Ceph v12, I'd have stayed that 
way.

> I also wonder why filestore could be that much faster, is this not 
> something else? Maybe some dangerous caching method was on?

My understanding is that Bluestore does not benefit from the Linux 
kernel filesystem cache.  On paper, Bluestore *should* be faster, but 
it's hard to know for sure.

Maybe I should try migrating back to Filestore and see if that improves 
things?
--
Stuart Longland (aka Redhatter, VK4MSL)

I haven't lost my mind...
  ...it's backed up on a tape somewhere.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Future of Filestore?

[Marc Roos]

 
Maybe a bit of topic, just curious what speeds did you get previously? 
Depending on how you test your native drive of 5400rpm, the performance 
could be similar. 4k random read of my 7200rpm/5400 rpm results in 
~60iops at 260kB/s.
I also wonder why filestore could be that much faster, is this not 
something else? Maybe some dangerous caching method was on?



-Original Message-
From: Stuart Longland [mailto:stua...@longlandclan.id.au] 
Sent: vrijdag 19 juli 2019 12:22
To: ceph-users
Subject: [ceph-users] Future of Filestore?

Hi all,

Earlier this year, I did a migration from Ceph 10 to 12.  Previously, I 
was happily running Ceph v10 on Filestore with BTRFS, and getting 
reasonable performance.

Moving to Ceph v12 necessitated a migration away from this set-up, and 
reading the documentation, Bluestore seemed to be "the way", so a hasty 
migration was performed and now my then 3-node cluster moved to 
Bluestore.  I've since added two new nodes to that cluster and replaced 
the disks in all systems, so I have 5 WD20SPZX-00Us storing my data.

I'm now getting about 5MB/sec I/O speeds in my VMs.

I'm contemplating whether I migrate back to using Filestore (on XFS this 
time, since BTRFS appears to be a rude word despite Ceph v10 docs 
suggesting it as a good option), but I'm not sure what the road map is 
for supporting Filestore long-term.

Is Filestore likely to have long term support for the next few years or 
should I persevere with tuning Bluestore to get something that won't be 
outperformed by an early 90s PIO mode 0 IDE HDD?
--
Stuart Longland (aka Redhatter, VK4MSL)

I haven't lost my mind...
  ...it's backed up on a tape somewhere.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs snapshot scripting questions

[Marc Roos]

H, ok ok, test it first, can't remember if it is finished. Checks 
also if it is usefull to create a snapshot, by checking the size of the 
directory.

[@ cron.daily]# cat backup-archive-mail.sh
#!/bin/bash


cd /home/

for account in `ls -c1 /home/mail-archive/ | sort`
do
  /usr/local/sbin/backup-snap.sh mail-archive/$account 7
  /usr/local/sbin/backup-snap.sh archiveindex/$account 7
done 



[@ cron.daily]# cat /usr/local/sbin/backup-snap.sh
#!/bin/bash
#
# usage: ./backup-snap.sh dirname ret
#
# dont forget to change
# START_HOURS_RANGE=0-22
#
# command to backup:


# static variables
# BACKUPDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
BACKUPDIR="$PWD"
BACKUPPREFIX="snap"
CEPHSDIR=".snap"
ARCHIVEDAYS=30
DOW=`date +%u`
DOM=`date +%-d`

if [ $# -lt 2 ]
then
 echo ""
 echo "usage: $0 dirname ret-days "
 echo ""
 echo ""
exit;
fi

# used for hardlinking with rsync to previous version
PREVDAY=
# current version backup location
VB=
# argument variables
ARGS=$*
SNAPBAK=$1
BACKUPDAYS=$2
ARCHIVE=0

function setpreviousday {

PREVDAY=`expr $DOM - 1`

if [ $PREVDAY -eq 0 ]
then
  PREVDAY=$BACKUPDAYS
fi
}

function getdirsize {
local path="$1"
local size=0
local rval=0

size=$(getfattr --only-values --absolute-names -d -m ceph.dir.rbytes 
"$path" 2> /dev/null)
if [ $? -eq 0 ]
then
  rval=$size
fi

echo -n $rval
}

function resetdom {
# reset the dom counter so XX days are backuped

if [ $DOM -gt $BACKUPDAYS ]
then
  DOM=`expr $DOM - $BACKUPDAYS`
  resetdom
fi

}

function removewhitespace {
NEW=`echo -n "$1" | sed -e 's/\s//g' `
echo -n "$NEW"
}


function createsnapshot () {
  SNAPBAK="$1"
  SNNAME="$2"

  mkdir "$BACKUPDIR/$SNAPBAK/$CEPHSDIR/$SNNAME"
}

function snapshotremove () {
  SNAPBAK="$1"
  SNNAME="$2"

  #echo "removing snapshot $SNAPBAK"
  rmdir "$BACKUPDIR/$SNAPBAK/$CEPHSDIR/$SNNAME"
  if [ $? -ne 0 ]
  then
echo "error removing old snapshot $SNAPBAK"
exit;
  fi
  sleep 2
}

function snapshotexists () {
# returns 0 if exists
  rval=1
  SNAPBAK="$1"
  SNNAME="$2"

  FOUND="$BACKUPDIR/$SNAPBAK/$CEPHSDIR/$SNNAME"
  if [ -d $FOUND ]
  then
rval=0
  fi

  echo -n $rval
}


# script arguments
#setargs

umask 0027


# reset day of month in scale of 1 to BACKUPDAYS
resetdom
VB=$DOM

# calculate previous day number
setpreviousday

# do server backups
SNNAME=$BACKUPPREFIX"-"$VB


# do only snapshot if there is data
if [ $(getdirsize "$SNAPBAK") -ne 0 ]
then
  if [ $(snapshotexists $SNAPBAK $SNNAME) -eq 0 ]
  then
snapshotremove $SNAPBAK $SNNAME
  fi

  createsnapshot $SNAPBAK $SNNAME
fi





-Original Message-
From: Robert Ruge [mailto:robert.r...@deakin.edu.au] 
Sent: woensdag 17 juli 2019 2:44
To: ceph-users@lists.ceph.com
Subject: [ceph-users] cephfs snapshot scripting questions

Greetings.

 

Before I reinvent the wheel has anyone written a script to maintain X 
number of snapshots on a cephfs file system that can be run through 
cron?

I am aware of the cephfs-snap code but just wondering if there are any 
other options out there.

 

On a related note which of these options would be better?

1.   Maintain one .snap directory at the root of the cephfs tree - 
/ceph/.snap

2.   Have a .snap directory for every second level directory 
/ceph/user/.snap

 

I am thinking the later might make it more obvious for the users to do 
their own restores but wondering what the resource implications of 
either approach might be.

 

The documentation indicates that I should use kernel >= 4.17 for cephfs. 
 I’m currently using Mimic 13.2.6 on Ubuntu 18.04 with kernel version 
4.15.0. What issues might I see with this combination? I’m hesitant to 
upgrade to an unsupported kernel on Ubuntu but wondering if I’m going 
to be playing Russian Roulette with this combo. 

 

Are there any gotcha’s I should be aware of before plunging into full 
blown cephfs snapshotting?

 

Regards and thanks.

Robert Ruge

 


Important Notice: The contents of this email are intended solely for the 
named addressee and are confidential; any unauthorised use, reproduction 
or storage of the contents is expressly prohibited. If you have received 
this email in error, please delete it and any attachments immediately 
and advise the sender by return email or telephone.

Deakin University does not warrant that this email and any attachments 
are error or virus free. 


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] New best practices for osds???

[Marc Roos]

 

With the sas expander you are putting more drives on 'one port'. Just 
make sure you do not create a bottle neck there, adding to many drives. 
I guess this depends on the speed of the drives. Then you should be fine 
not?




-Original Message-
From: Stolte, Felix [mailto:f.sto...@fz-juelich.de] 
Sent: dinsdag 16 juli 2019 17:42
To: ceph-users
Subject: [ceph-users] New best practices for osds???

Hi guys,

our ceph cluster is performing way less than it could, based on the 
disks we are using. We could narrow it down to the storage controller 
(LSI SAS3008 HBA) in combination with an SAS expander. Yesterday we had 
a meeting with our hardware reseller and sale representatives of the 
hardware manufacturer to resolve the issue.

They told us, that "best practices" for ceph would be to deploy disks as 
Raid 0 consisting of one disk using a raid controller with a big 
writeback cache. 

Since this "best practice" is new to me, I would like to hear your 
opinion on this topic.

Regards Felix


-

-
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), 
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. 
Dr. Sebastian M. Schmidt

-

-
 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] "session established", "io error", "session lost, hunting for new mon" solution/fix

[Marc Roos]

 
Can you give me a little pointer on how and where to do the verbose 
logging? I am really interested in discovery why I am getting the large 
osdmap message. Maybe it is because of the snapshot being created?



-Original Message-
From: Ilya Dryomov [mailto:idryo...@gmail.com] 
Cc: paul.emmerich; ceph-users
Subject: Re: [ceph-users] "session established", "io error", "session 
lost, hunting for new mon" solution/fix

On Fri, Jul 12, 2019 at 5:38 PM Marc Roos  
wrote:
>
>
> Thanks Ilya for explaining. Am I correct to understand from the 
> link[0] mentioned in the issue, that because eg. I have an unhealthy 
> state for some time (1 pg on a insignificant pool) I have larger 
> osdmaps, triggering this issue? Or is just random bad luck? (Just a 
> bit curious why I have this issue)
>
> [0]
> https://www.mail-archive.com/ceph-users@lists.ceph.com/msg51522.html

I'm not sure.  I wouldn't expect one unhealthy PG to trigger a large 
osdmap message.  Only verbose logs can tell.

Thanks,

Ilya


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Returning to the performance in a small cluster topic

[Marc Roos]

 

Isn't that why you suppose to test up front? So you do not have shocking 
surprises? You can find in the mailing list archives some performance 
references also. 
I think it would be good to publish some performance results on the 
ceph.com website. Can’t be to difficult to put some default scenarios, 
used hardware and performance there in some nice graphs. I take it some 
here would be willing to contribute test results of their 
test/production clusters. This way new ceph’ers know what to expect 
from similar setups.



-Original Message-
From: Jordan Share [mailto:readm...@krotus.com] 
Sent: maandag 15 juli 2019 20:16
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Returning to the performance in a small 
cluster topic

We found shockingly bad committed IOPS/latencies on ceph.

We could get roughly 20-30 IOPS when running this fio invocation from 
within a vm:
fio --name=seqwrite --rw=write --direct=1 --ioengine=libaio --bs=32k
--numjobs=1 --size=2G --runtime=60 --group_reporting --fsync=1

For non-committed IO, we get about 2800 iops, with this invocation:
fio --name=seqwrite --rw=write --direct=1 --ioengine=libaio --bs=32k
--numjobs=1 --size=2G --runtime=150 --group_reporting

So, maybe, if PostreSQL has a lot of committed IO needs, you might not 
have the performance you're expecting.

You could try running your fio tests with "--fsync=1" and see if those 
numbers (which I'd expect to be very low) would be in line with your 
PostgreSQL performance.

Jordan


On 7/15/2019 7:08 AM, Drobyshevskiy, Vladimir wrote:
> Dear colleagues,
> 
>    I would like to ask you for help with a performance problem on a 
> site backed with ceph storage backend. Cluster details below.
> 
>    I've got a big problem with PostgreSQL performance. It runs inside 
> a VM with virtio-scsi ceph rbd image. And I see constant ~100% disk 
> load with up to hundreds milliseconds latencies (via atop) even when 
> pg_top shows 10-20 tps. All other resources are almost untouched - 
> there is a lot of memory and free CPU cores, DB fits memory but still 
> has performance issues.
> 
>    The cluster itself:
>    nautilus
>    6 nodes, 7 SSD with 2 OSDs per SSD (14 OSDs in overall).
>    Each node: 2x Intel Xeon E5-2665 v1 (governor = performance, 
> powersaving disabled), 64GB RAM, Samsung SM863 1.92TB SSD, QDR 
Infiniband.
> 
>    I've made fio benchmarking with three type of measures:
>    a VM with virtio-scsi driver,
>    baremetal host with mounted rbd image
>    and the same baremetal host with mounted lvm partition on SM863 SSD 

> drive.
> 
>    I've set bs=8k (as Postgres writes 8k blocks) and tried 1 and 8 
jobs.
> 
>    Here are some results: https://pastebin.com/TFUg5fqA
>    Drives load on the OSD hosts are very low, just a few percent.
> 
>    Here is my ceph config: https://pastebin.com/X5ZwaUrF
> 
>    Numbers don't look very good from my point of view but they are 
> also not really bad (are they?). But I don't really know the next 
> direction I can go to solve the problem with PostgreSQL.
> 
>    I've tried to make an RAID0 with mdraid and 2 virtual drives but 
> haven't noticed any difference.
> 
>    Could you please tell me:
>    Are these performance numbers good or bad according to the 
hardware?
>    Is it possible to tune anything more? May be you can point me to 
> docs or other papers?
>    Does any special VM tuning for the PostgreSQL\ceph cooperation 
exist?
>    Thank you in advance!
> 
> --
> Best regards,
> Vladimir
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] "session established", "io error", "session lost, hunting for new mon" solution/fix

[Marc Roos]

 
Thanks Ilya for explaining. Am I correct to understand from the link[0] 
mentioned in the issue, that because eg. I have an unhealthy state for 
some time (1 pg on a insignificant pool) I have larger osdmaps, 
triggering this issue? Or is just random bad luck? (Just a bit curious 
why I have this issue)

[0]
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg51522.html

-Original Message-
Subject: Re: [ceph-users] "session established", "io error", "session 
lost, hunting for new mon" solution/fix

On Fri, Jul 12, 2019 at 12:33 PM Paul Emmerich  
wrote:
>
>
>
> On Thu, Jul 11, 2019 at 11:36 PM Marc Roos  
wrote:
>> Anyone know why I would get these? Is it not strange to get them in a 

>> 'standard' setup?
>
> you are probably running on an ancient kernel. this bug has been fixed 
a long time ago.

This is not a kernel bug:

http://tracker.ceph.com/issues/38040

It is possible to hit with few OSDs too.  The actual problem is the size 
of the osdmap message which can contain multiple full osdmaps, not the 
number of OSDs.  The size of a full osdmap is proportional to the number 
of OSDs but it's not the only way to get a big osdmap message.

As you have experienced, these settings used to be expressed in the 
number of osdmaps and our defaults were too high for a stream of full 
osdmaps (as opposed to incrementals).  It is now expressed in bytes, the 
patch should be in 12.2.13.

>
>> -Original Message-
>> Subject: [ceph-users] "session established", "io error", "session 
>> lost, hunting for new mon" solution/fix
>>
>>
>> I have on a cephfs client again (luminous cluster, centos7, only 32 
>> osds!). Wanted to share the 'fix'
>>
>> [Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 session 
>> established [Thu Jul 11 12:16:09 2019] libceph: mon0 
>> 192.168.10.111:6789 io error [Thu Jul 11 12:16:09 2019] libceph: mon0 

>> 192.168.10.111:6789 session lost, hunting for new mon [Thu Jul 11 
>> 12:16:09 2019] libceph: mon2 192.168.10.113:6789 session established 
>> [Thu Jul 11 12:16:09 2019] libceph: mon2 192.168.10.113:6789 io error 

>> [Thu Jul 11 12:16:09 2019] libceph: mon2 192.168.10.113:6789 session 
>> lost, hunting for new mon [Thu Jul 11 12:16:09 2019] libceph: mon0 
>> 192.168.10.111:6789 session established [Thu Jul 11 12:16:09 2019] 
>> libceph: mon0 192.168.10.111:6789 io error [Thu Jul 11 12:16:09 2019] 

>> libceph: mon0 192.168.10.111:6789 session lost, hunting for new mon 
>> [Thu Jul 11 12:16:09 2019] libceph: mon1 192.168.10.112:6789 session 
>> established [Thu Jul 11 12:16:09 2019] libceph: mon1 
>> 192.168.10.112:6789 io error [Thu Jul 11 12:16:09 2019] libceph: mon1 

>> 192.168.10.112:6789 session lost, hunting for new mon
>>
>> 1) I blocked client access to the monitors with iptables -I INPUT -p 
>> tcp -s 192.168.10.43 --dport 6789 -j REJECT Resulting in
>>
>> [Thu Jul 11 12:34:16 2019] libceph: mon1 192.168.10.112:6789 socket 
>> closed (con state CONNECTING) [Thu Jul 11 12:34:18 2019] libceph: 
>> mon1 192.168.10.112:6789 socket closed (con state CONNECTING) [Thu 
>> Jul 11 12:34:22 2019] libceph: mon1 192.168.10.112:6789 socket closed 

>> (con state CONNECTING) [Thu Jul 11 12:34:26 2019] libceph: mon2 
>> 192.168.10.113:6789 socket closed (con state CONNECTING) [Thu Jul 11 
>> 12:34:27 2019] libceph: mon2 192.168.10.113:6789 socket closed (con 
>> state CONNECTING) [Thu Jul 11 12:34:28 2019] libceph: mon2 
>> 192.168.10.113:6789 socket closed (con state CONNECTING) [Thu Jul 11 
>> 12:34:30 2019] libceph: mon1 192.168.10.112:6789 socket closed (con 
>> state CONNECTING) [Thu Jul 11 12:34:30 2019] libceph: mon2 
>> 192.168.10.113:6789 socket closed (con state CONNECTING) [Thu Jul 11 
>> 12:34:34 2019] libceph: mon2 192.168.10.113:6789 socket closed (con 
>> state CONNECTING) [Thu Jul 11 12:34:42 2019] libceph: mon2 
>> 192.168.10.113:6789 socket closed (con state CONNECTING) [Thu Jul 11 
>> 12:34:44 2019] libceph: mon0 192.168.10.111:6789 socket closed (con 
>> state CONNECTING) [Thu Jul 11 12:34:45 2019] libceph: mon0 
>> 192.168.10.111:6789 socket closed (con state CONNECTING) [Thu Jul 11 
>> 12:34:46 2019] libceph: mon0 192.168.10.111:6789 socket closed (con 
>> state CONNECTING)
>>
>> 2) I applied the suggested changes to the osd map message max, 
>> mentioned
>>
>> in early threads[0]
>> ceph tell osd.* injectargs '--osd_map_message_max=10'
>> ceph tell mon.* injectargs '--osd_map_message_max=10'
>> [@c01 ~]# ceph daemon osd.0 config show|grep message_max
>> "osd_map_message_max": "10",
>

Re: [ceph-users] "session established", "io error", "session lost, hunting for new mon" solution/fix

[Marc Roos]

 
Paul, this should have been/is back ported to this kernel not?


-Original Message-
From: Paul Emmerich [mailto:paul.emmer...@croit.io] 
Cc: ceph-users
Subject: Re: [ceph-users] "session established", "io error", "session 
lost, hunting for new mon" solution/fix

 

Anyone know why I would get these? Is it not strange to get them in 
a 
'standard' setup?



you are probably running on an ancient kernel. this bug has been fixed a 
long time ago.


Paul

 






-Original Message-
Subject: [ceph-users] "session established", "io error", "session 
lost, 
hunting for new mon" solution/fix


I have on a cephfs client again (luminous cluster, centos7, only 32 

osds!). Wanted to share the 'fix'

[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 
session 
established
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 io 
error
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 
session 
lost, hunting for new mon
[Thu Jul 11 12:16:09 2019] libceph: mon2 192.168.10.113:6789 
session 
established
[Thu Jul 11 12:16:09 2019] libceph: mon2 192.168.10.113:6789 io 
error
[Thu Jul 11 12:16:09 2019] libceph: mon2 192.168.10.113:6789 
session 
lost, hunting for new mon
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 
session 
established
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 io 
error
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 
session 
lost, hunting for new mon
[Thu Jul 11 12:16:09 2019] libceph: mon1 192.168.10.112:6789 
session 
established
[Thu Jul 11 12:16:09 2019] libceph: mon1 192.168.10.112:6789 io 
error
[Thu Jul 11 12:16:09 2019] libceph: mon1 192.168.10.112:6789 
session 
lost, hunting for new mon

1) I blocked client access to the monitors with
iptables -I INPUT -p tcp -s 192.168.10.43 --dport 6789 -j REJECT
Resulting in 

[Thu Jul 11 12:34:16 2019] libceph: mon1 192.168.10.112:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:18 2019] libceph: mon1 192.168.10.112:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:22 2019] libceph: mon1 192.168.10.112:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:26 2019] libceph: mon2 192.168.10.113:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:27 2019] libceph: mon2 192.168.10.113:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:28 2019] libceph: mon2 192.168.10.113:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:30 2019] libceph: mon1 192.168.10.112:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:30 2019] libceph: mon2 192.168.10.113:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:34 2019] libceph: mon2 192.168.10.113:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:42 2019] libceph: mon2 192.168.10.113:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:44 2019] libceph: mon0 192.168.10.111:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:45 2019] libceph: mon0 192.168.10.111:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:46 2019] libceph: mon0 192.168.10.111:6789 socket 

closed (con state CONNECTING)

2) I applied the suggested changes to the osd map message max, 
mentioned 

in early threads[0]
ceph tell osd.* injectargs '--osd_map_message_max=10'
ceph tell mon.* injectargs '--osd_map_message_max=10'
[@c01 ~]# ceph daemon osd.0 config show|grep message_max
"osd_map_message_max": "10",
[@c01 ~]# ceph daemon mon.a config show|grep message_max
"osd_map_message_max": "10",

[0]
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg54419.htm
l
http://tracker.ceph.com/issues/38040

3) Allow access to a monitor with
iptables -D INPUT -p tcp -s 192.168.10.43 --dport 6789 -j REJECT

Getting 
[Thu Jul 11 12:39:26 2019] libceph: mon0 192.168.10.111:6789 
session 
established
[Thu Jul 11 12:39:26 2019] libceph: osd0 down
[Thu Jul 11 12:39:26 2019] libceph: osd0 up

Problems solved, in D state hung unmount was released. 

I am not sure if the prolonged disconnection to the monitors was 
the 
solution or the osd_map_message_max=10, or both. 





___
ceph-users mailing list

Re: [ceph-users] "session established", "io error", "session lost, hunting for new mon" solution/fix

[Marc Roos]

 

 
Hi Paul, 

Thanks for your reply, I am running 3.10.0-957.12.2.el7.x86_64, it is 
from may 2019.



-Original Message-
From: Paul Emmerich [mailto:paul.emmer...@croit.io] 
Sent: vrijdag 12 juli 2019 12:34
To: Marc Roos
Cc: ceph-users
Subject: Re: [ceph-users] "session established", "io error", "session 
lost, hunting for new mon" solution/fix



On Thu, Jul 11, 2019 at 11:36 PM Marc Roos  
wrote:


 

Anyone know why I would get these? Is it not strange to get them in 
a 
'standard' setup?



you are probably running on an ancient kernel. this bug has been fixed a 
long time ago.


Paul

 






-Original Message-
Subject: [ceph-users] "session established", "io error", "session 
lost, 
hunting for new mon" solution/fix


I have on a cephfs client again (luminous cluster, centos7, only 32 

osds!). Wanted to share the 'fix'

[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 
session 
established
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 io 
error
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 
session 
lost, hunting for new mon
[Thu Jul 11 12:16:09 2019] libceph: mon2 192.168.10.113:6789 
session 
established
[Thu Jul 11 12:16:09 2019] libceph: mon2 192.168.10.113:6789 io 
error
[Thu Jul 11 12:16:09 2019] libceph: mon2 192.168.10.113:6789 
session 
lost, hunting for new mon
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 
session 
established
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 io 
error
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 
session 
lost, hunting for new mon
[Thu Jul 11 12:16:09 2019] libceph: mon1 192.168.10.112:6789 
session 
established
[Thu Jul 11 12:16:09 2019] libceph: mon1 192.168.10.112:6789 io 
error
[Thu Jul 11 12:16:09 2019] libceph: mon1 192.168.10.112:6789 
session 
lost, hunting for new mon

1) I blocked client access to the monitors with
iptables -I INPUT -p tcp -s 192.168.10.43 --dport 6789 -j REJECT
Resulting in 

[Thu Jul 11 12:34:16 2019] libceph: mon1 192.168.10.112:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:18 2019] libceph: mon1 192.168.10.112:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:22 2019] libceph: mon1 192.168.10.112:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:26 2019] libceph: mon2 192.168.10.113:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:27 2019] libceph: mon2 192.168.10.113:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:28 2019] libceph: mon2 192.168.10.113:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:30 2019] libceph: mon1 192.168.10.112:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:30 2019] libceph: mon2 192.168.10.113:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:34 2019] libceph: mon2 192.168.10.113:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:42 2019] libceph: mon2 192.168.10.113:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:44 2019] libceph: mon0 192.168.10.111:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:45 2019] libceph: mon0 192.168.10.111:6789 socket 

closed (con state CONNECTING)
[Thu Jul 11 12:34:46 2019] libceph: mon0 192.168.10.111:6789 socket 

closed (con state CONNECTING)

2) I applied the suggested changes to the osd map message max, 
mentioned 

in early threads[0]
ceph tell osd.* injectargs '--osd_map_message_max=10'
ceph tell mon.* injectargs '--osd_map_message_max=10'
[@c01 ~]# ceph daemon osd.0 config show|grep message_max
"osd_map_message_max": "10",
[@c01 ~]# ceph daemon mon.a config show|grep message_max
"osd_map_message_max": "10",

[0]
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg54419.htm
l
http://tracker.ceph.com/issues/38040

3) Allow access to a monitor with
iptables -D INPUT -p tcp -s 192.168.10.43 --dport 6789 -j REJECT

Getting 
[Thu Jul 11 12:39:26 2019] libceph: mon0 192.168.10.111:6789 
session 
established
[Thu Jul 11 12:39:26 2019] libceph: osd0 down
[Thu Jul 11 12:39:26 2019] libceph: osd0 up

Problems solved, in D state hung unmount was released. 

I am not sure if the prolo

Re: [ceph-users] "session established", "io error", "session lost, hunting for new mon" solution/fix

[Marc Roos]

 

Anyone know why I would get these? Is it not strange to get them in a 
'standard' setup?





-Original Message-
Subject: [ceph-users] "session established", "io error", "session lost, 
hunting for new mon" solution/fix


I have on a cephfs client again (luminous cluster, centos7, only 32 
osds!). Wanted to share the 'fix'

[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 session 
established
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 io error
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 session 
lost, hunting for new mon
[Thu Jul 11 12:16:09 2019] libceph: mon2 192.168.10.113:6789 session 
established
[Thu Jul 11 12:16:09 2019] libceph: mon2 192.168.10.113:6789 io error
[Thu Jul 11 12:16:09 2019] libceph: mon2 192.168.10.113:6789 session 
lost, hunting for new mon
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 session 
established
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 io error
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 session 
lost, hunting for new mon
[Thu Jul 11 12:16:09 2019] libceph: mon1 192.168.10.112:6789 session 
established
[Thu Jul 11 12:16:09 2019] libceph: mon1 192.168.10.112:6789 io error
[Thu Jul 11 12:16:09 2019] libceph: mon1 192.168.10.112:6789 session 
lost, hunting for new mon

1) I blocked client access to the monitors with
iptables -I INPUT -p tcp -s 192.168.10.43 --dport 6789 -j REJECT
Resulting in 

[Thu Jul 11 12:34:16 2019] libceph: mon1 192.168.10.112:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:18 2019] libceph: mon1 192.168.10.112:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:22 2019] libceph: mon1 192.168.10.112:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:26 2019] libceph: mon2 192.168.10.113:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:27 2019] libceph: mon2 192.168.10.113:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:28 2019] libceph: mon2 192.168.10.113:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:30 2019] libceph: mon1 192.168.10.112:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:30 2019] libceph: mon2 192.168.10.113:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:34 2019] libceph: mon2 192.168.10.113:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:42 2019] libceph: mon2 192.168.10.113:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:44 2019] libceph: mon0 192.168.10.111:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:45 2019] libceph: mon0 192.168.10.111:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:46 2019] libceph: mon0 192.168.10.111:6789 socket 
closed (con state CONNECTING)

2) I applied the suggested changes to the osd map message max, mentioned 

in early threads[0]
ceph tell osd.* injectargs '--osd_map_message_max=10'
ceph tell mon.* injectargs '--osd_map_message_max=10'
[@c01 ~]# ceph daemon osd.0 config show|grep message_max
"osd_map_message_max": "10",
[@c01 ~]# ceph daemon mon.a config show|grep message_max
"osd_map_message_max": "10",

[0]
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg54419.html
http://tracker.ceph.com/issues/38040

3) Allow access to a monitor with
iptables -D INPUT -p tcp -s 192.168.10.43 --dport 6789 -j REJECT

Getting 
[Thu Jul 11 12:39:26 2019] libceph: mon0 192.168.10.111:6789 session 
established
[Thu Jul 11 12:39:26 2019] libceph: osd0 down
[Thu Jul 11 12:39:26 2019] libceph: osd0 up

Problems solved, in D state hung unmount was released. 

I am not sure if the prolonged disconnection to the monitors was the 
solution or the osd_map_message_max=10, or both. 





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Libceph clock drift or I guess kernel clock drift issue

[Marc Roos]

I noticed that the dmesg -T gives incorrect time, the messages have a 
time in the future compared to the system time. Not sure if this is 
libceph issue or a kernel issue.

[Thu Jul 11 10:41:22 2019] libceph: mon2 192.168.10.113:6789 session 
lost, hunting for new mon
[Thu Jul 11 10:41:22 2019] libceph: osd22 192.168.10.111:6811 io error
[Thu Jul 11 10:41:22 2019] libceph: mon1 192.168.10.112:6789 session 
established
[Thu Jul 11 10:41:22 2019] libceph: mon1 192.168.10.112:6789 io error
[Thu Jul 11 10:41:22 2019] libceph: mon1 192.168.10.112:6789 session 
lost, hunting for new mon
[Thu Jul 11 10:41:22 2019] libceph: mon0 192.168.10.111:6789 session 
established


[@ ]# uptime
 10:39:17 up 50 days, 13:31,  2 users,  load average: 3.60, 3.02, 2.57


[@~]# uname -a
Linux c01 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 21:24:32 UTC 2019 
x86_64 x86_64 x86_64 GNU/Linux
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] "session established", "io error", "session lost, hunting for new mon" solution/fix

[Marc Roos]

I have on a cephfs client again (luminous cluster, centos7, only 32 
osds!). Wanted to share the 'fix'

[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 session 
established
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 io error
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 session 
lost, hunting for new mon
[Thu Jul 11 12:16:09 2019] libceph: mon2 192.168.10.113:6789 session 
established
[Thu Jul 11 12:16:09 2019] libceph: mon2 192.168.10.113:6789 io error
[Thu Jul 11 12:16:09 2019] libceph: mon2 192.168.10.113:6789 session 
lost, hunting for new mon
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 session 
established
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 io error
[Thu Jul 11 12:16:09 2019] libceph: mon0 192.168.10.111:6789 session 
lost, hunting for new mon
[Thu Jul 11 12:16:09 2019] libceph: mon1 192.168.10.112:6789 session 
established
[Thu Jul 11 12:16:09 2019] libceph: mon1 192.168.10.112:6789 io error
[Thu Jul 11 12:16:09 2019] libceph: mon1 192.168.10.112:6789 session 
lost, hunting for new mon

1) I blocked client access to the monitors with
iptables -I INPUT -p tcp -s 192.168.10.43 --dport 6789 -j REJECT
Resulting in 

[Thu Jul 11 12:34:16 2019] libceph: mon1 192.168.10.112:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:18 2019] libceph: mon1 192.168.10.112:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:22 2019] libceph: mon1 192.168.10.112:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:26 2019] libceph: mon2 192.168.10.113:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:27 2019] libceph: mon2 192.168.10.113:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:28 2019] libceph: mon2 192.168.10.113:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:30 2019] libceph: mon1 192.168.10.112:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:30 2019] libceph: mon2 192.168.10.113:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:34 2019] libceph: mon2 192.168.10.113:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:42 2019] libceph: mon2 192.168.10.113:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:44 2019] libceph: mon0 192.168.10.111:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:45 2019] libceph: mon0 192.168.10.111:6789 socket 
closed (con state CONNECTING)
[Thu Jul 11 12:34:46 2019] libceph: mon0 192.168.10.111:6789 socket 
closed (con state CONNECTING)

2) I applied the suggested changes to the osd map message max, mentioned 
in early threads[0]
ceph tell osd.* injectargs '--osd_map_message_max=10'
ceph tell mon.* injectargs '--osd_map_message_max=10'
[@c01 ~]# ceph daemon osd.0 config show|grep message_max
"osd_map_message_max": "10",
[@c01 ~]# ceph daemon mon.a config show|grep message_max
"osd_map_message_max": "10",

[0]
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg54419.html
http://tracker.ceph.com/issues/38040

3) Allow access to a monitor with
iptables -D INPUT -p tcp -s 192.168.10.43 --dport 6789 -j REJECT

Getting 
[Thu Jul 11 12:39:26 2019] libceph: mon0 192.168.10.111:6789 session 
established
[Thu Jul 11 12:39:26 2019] libceph: osd0 down
[Thu Jul 11 12:39:26 2019] libceph: osd0 up

Problems solved, in D state hung unmount was released. 

I am not sure if the prolonged disconnection to the monitors was the 
solution or the osd_map_message_max=10, or both. 





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] shutdown down all monitors

[Marc Roos]

Can I temporary shutdown all my monitors? This only affects new 
connections not? Existing will still keep running?



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Luminous cephfs maybe not to stable as expected?

[Marc Roos]

 


I decided to restart osd.0, then the load of the cephfs and on all osd 
nodes dropped. After this I still have on the first server


[@~]# cat 
/sys/kernel/debug/ceph/0f1701f5-453a-4a3b-928d-f652a2bbbcb0.client357431
0/osdc
REQUESTS 0 homeless 0
LINGER REQUESTS
BACKOFFS


[@~]# cat 
/sys/kernel/debug/ceph/0f1701f5-453a-4a3b-928d-f652a2bbbcb0.client358422
4/osdc
REQUESTS 2 homeless 0
317841  osd020.d6ec44c1 20.1[0,28,5]/0  [0,28,5]/0  
e65040  10001b44a70.0x40001c102023  read
317853  osd020.5956d31b 20.1b   [0,5,10]/0  [0,5,10]/0  
e65040  10001ad8962.0x40001c40731   read
LINGER REQUESTS
BACKOFFS

And dmesg -T keeps giving me these (again with wrong timestamps)

[Thu Jul 11 11:23:21 2019] libceph: mon1 192.168.10.112:6789 session 
established
[Thu Jul 11 11:23:21 2019] libceph: mon1 192.168.10.112:6789 io error
[Thu Jul 11 11:23:21 2019] libceph: mon1 192.168.10.112:6789 session 
lost, hunting for new mon
[Thu Jul 11 11:23:21 2019] libceph: mon0 192.168.10.111:6789 session 
established
[Thu Jul 11 11:23:21 2019] libceph: mon0 192.168.10.111:6789 io error
[Thu Jul 11 11:23:21 2019] libceph: mon0 192.168.10.111:6789 session 
lost, hunting for new mon
[Thu Jul 11 11:23:21 2019] libceph: mon2 192.168.10.113:6789 session 
established
[Thu Jul 11 11:23:21 2019] libceph: mon2 192.168.10.113:6789 io error
[Thu Jul 11 11:23:21 2019] libceph: mon2 192.168.10.113:6789 session 
lost, hunting for new mon
[Thu Jul 11 11:23:21 2019] libceph: mon1 192.168.10.112:6789 session 
established
[Thu Jul 11 11:23:21 2019] libceph: mon1 192.168.10.112:6789 io error
[Thu Jul 11 11:23:21 2019] libceph: mon1 192.168.10.112:6789 session 
lost, hunting for new mon
[Thu Jul 11 11:23:21 2019] libceph: mon0 192.168.10.111:6789 session 
established
[Thu Jul 11 11:23:21 2019] libceph: mon0 192.168.10.111:6789 io error
[Thu Jul 11 11:23:21 2019] libceph: mon0 192.168.10.111:6789 session 
lost, hunting for new mon

What to do now? Restarting the monitor did not help.


-Original Message-
Subject: Re: [ceph-users] Luminous cephfs maybe not to stable as 
expected?

 

Forgot to add these

[@ ~]# cat
/sys/kernel/debug/ceph/0f1701f5-453a-4a3b-928d-f652a2bbbcb0.client357431
0/osdc
REQUESTS 0 homeless 0
LINGER REQUESTS
BACKOFFS

[@~]# cat
/sys/kernel/debug/ceph/0f1701f5-453a-4a3b-928d-f652a2bbbcb0.client358422
4/osdc
REQUESTS 38 homeless 0
317841  osd020.d6ec44c1 20.1[0,28,5]/0  [0,28,5]/0  
e65040  10001b44a70.0x40001c101139  read
317853  osd020.5956d31b 20.1b   [0,5,10]/0  [0,5,10]/0  
e65040  10001ad8962.0x40001c39847   read
317835  osd320.ede889de 20.1e   [3,12,27]/3 [3,12,27]/3 
e65040  10001ad80f6.0x40001c87758   read
317838  osd320.7b730a4e 20.e[3,31,9]/3  [3,31,9]/3  
e65040  10001ad89d8.0x40001c83444   read
317844  osd320.feead84c 20.c[3,13,18]/3 [3,13,18]/3 
e65040  10001ad8733.0x40001c77267   read
317852  osd320.bd2658e  20.e[3,31,9]/3  [3,31,9]/3  
e65040  10001ad7e00.0x40001c39331   read
317830  osd420.922e6d04 20.4[4,16,27]/4 [4,16,27]/4 
e65040  10001ad80f2.0x40001c86326   read
317837  osd420.fe93d4ab 20.2b   [4,14,25]/4 [4,14,25]/4 
e65040  10001ad80fb.0x40001c78951   read
317839  osd420.d7af926b 20.2b   [4,14,25]/4 [4,14,25]/4 
e65040  10001ad80ee.0x40001c77556   read
317849  osd520.5fcb95c5 20.5[5,18,29]/5 [5,18,29]/5 
e65040  10001ad7f75.0x40001c61147   read
317857  osd520.28764e9a 20.1a   [5,7,28]/5  [5,7,28]/5  
e65040  10001ad8a10.0x40001c30369   read
317859  osd520.7bb79985 20.5[5,18,29]/5 [5,18,29]/5 
e65040  10001ad7fe8.0x40001c27942   read
317836  osd820.e7bf5cf4 20.34   [8,5,10]/8  [8,5,10]/8  
e65040  10001ad7d79.0x40001c133699  read
317842  osd820.abbb9df4 20.34   [8,5,10]/8  [8,5,10]/8  
e65040  10001d5903f.0x40001c125308  read
317850  osd820.ecd0034  20.34   [8,5,10]/8  [8,5,10]/8  
e65040  10001ad89b2.0x40001c68348   read
317854  osd820.cef50134 20.34   [8,5,10]/8  [8,5,10]/8  
e65040  10001ad8728.0x40001c57431   read
317861  osd820.3e859bb4 20.34   [8,5,10]/8  [8,5,10]/8  
e65040  10001ad8108.0x40001c50642   read
317847  osd920.fc9e9f43 20.3[9,29,17]/9 [9,29,17]/9 
e65040  10001ad8101.0x40001c88464   read
317848  osd920.d32b6ac3 20.3[9,29,17]/9 [9,29,17]/9 
e65040  10001ad8100.0x40001c85929   read
317862  osd11   

Re: [ceph-users] Luminous cephfs maybe not to stable as expected?

[Marc Roos]

 

Forgot to add these

[@ ~]# cat 
/sys/kernel/debug/ceph/0f1701f5-453a-4a3b-928d-f652a2bbbcb0.client357431
0/osdc
REQUESTS 0 homeless 0
LINGER REQUESTS
BACKOFFS

[@~]# cat 
/sys/kernel/debug/ceph/0f1701f5-453a-4a3b-928d-f652a2bbbcb0.client358422
4/osdc
REQUESTS 38 homeless 0
317841  osd020.d6ec44c1 20.1[0,28,5]/0  [0,28,5]/0  
e65040  10001b44a70.0x40001c101139  read
317853  osd020.5956d31b 20.1b   [0,5,10]/0  [0,5,10]/0  
e65040  10001ad8962.0x40001c39847   read
317835  osd320.ede889de 20.1e   [3,12,27]/3 [3,12,27]/3 
e65040  10001ad80f6.0x40001c87758   read
317838  osd320.7b730a4e 20.e[3,31,9]/3  [3,31,9]/3  
e65040  10001ad89d8.0x40001c83444   read
317844  osd320.feead84c 20.c[3,13,18]/3 [3,13,18]/3 
e65040  10001ad8733.0x40001c77267   read
317852  osd320.bd2658e  20.e[3,31,9]/3  [3,31,9]/3  
e65040  10001ad7e00.0x40001c39331   read
317830  osd420.922e6d04 20.4[4,16,27]/4 [4,16,27]/4 
e65040  10001ad80f2.0x40001c86326   read
317837  osd420.fe93d4ab 20.2b   [4,14,25]/4 [4,14,25]/4 
e65040  10001ad80fb.0x40001c78951   read
317839  osd420.d7af926b 20.2b   [4,14,25]/4 [4,14,25]/4 
e65040  10001ad80ee.0x40001c77556   read
317849  osd520.5fcb95c5 20.5[5,18,29]/5 [5,18,29]/5 
e65040  10001ad7f75.0x40001c61147   read
317857  osd520.28764e9a 20.1a   [5,7,28]/5  [5,7,28]/5  
e65040  10001ad8a10.0x40001c30369   read
317859  osd520.7bb79985 20.5[5,18,29]/5 [5,18,29]/5 
e65040  10001ad7fe8.0x40001c27942   read
317836  osd820.e7bf5cf4 20.34   [8,5,10]/8  [8,5,10]/8  
e65040  10001ad7d79.0x40001c133699  read
317842  osd820.abbb9df4 20.34   [8,5,10]/8  [8,5,10]/8  
e65040  10001d5903f.0x40001c125308  read
317850  osd820.ecd0034  20.34   [8,5,10]/8  [8,5,10]/8  
e65040  10001ad89b2.0x40001c68348   read
317854  osd820.cef50134 20.34   [8,5,10]/8  [8,5,10]/8  
e65040  10001ad8728.0x40001c57431   read
317861  osd820.3e859bb4 20.34   [8,5,10]/8  [8,5,10]/8  
e65040  10001ad8108.0x40001c50642   read
317847  osd920.fc9e9f43 20.3[9,29,17]/9 [9,29,17]/9 
e65040  10001ad8101.0x40001c88464   read
317848  osd920.d32b6ac3 20.3[9,29,17]/9 [9,29,17]/9 
e65040  10001ad8100.0x40001c85929   read
317862  osd11   20.ee6cc689 20.9[11,0,12]/11[11,0,12]/11
e65040  10001ad7d64.0x40001c40266   read
317843  osd12   20.a801f0e9 20.29   [12,26,8]/12[12,26,8]/12
e65040  10001ad7f07.0x40001c86610   read
317851  osd12   20.8bb48de9 20.29   [12,26,8]/12[12,26,8]/12
e65040  10001ad7e4f.0x40001c46746   read
317860  osd12   20.47815f36 20.36   [12,0,28]/12[12,0,28]/12
e65040  10001ad8035.0x40001c35249   read
317831  osd15   20.9e3acb53 20.13   [15,0,1]/15 [15,0,1]/15 
e65040  10001ad8978.0x40001c85329   read
317840  osd15   20.2a40efdf 20.1f   [15,4,17]/15[15,4,17]/15
e65040  10001ad7ef8.0x40001c76282   read
317846  osd15   20.8143f15f 20.1f   [15,4,17]/15[15,4,17]/15
e65040  10001ad89d1.0x40001c61297   read
317864  osd15   20.c889a49c 20.1c   [15,0,31]/15[15,0,31]/15
e65040  10001ad89fb.0x40001c24385   read
317832  osd18   20.f76227a  20.3a   [18,6,15]/18[18,6,15]/18
e65040  10001ad8020.0x40001c82852   read
317833  osd18   20.d8edab31 20.31   [18,29,14]/18   [18,29,14]/18   
e65040  10001ad8952.0x40001c82852   read
317858  osd18   20.8f69d231 20.31   [18,29,14]/18   [18,29,14]/18   
e65040  10001ad8176.0x40001c32400   read
317855  osd22   20.b3342c0f 20.f[22,18,31]/22   [22,18,31]/22   
e65040  10001ad8146.0x40001c51024   read
317863  osd23   20.cde0ce7b 20.3b   [23,1,6]/23 [23,1,6]/23 
e65040  10001ad856c.0x40001c34521   read
317865  osd23   20.702d2dfe 20.3e   [23,9,22]/23[23,9,22]/23
e65040  10001ad8a5e.0x40001c30664   read
317866  osd23   20.cb4a32fe 20.3e   [23,9,22]/23[23,9,22]/23
e65040  10001ad8575.0x40001c29683   read
317867  osd23   20.9a008910 20.10   [23,12,6]/23[23,12,6]/23
e65040  10001ad7d24.0x40001c29683   read
317834  osd25   20.6efd4911

[ceph-users] Luminous cephfs maybe not to stable as expected?

[Marc Roos]

Maybe this requires some attention. I have a default centos7 (maybe not 
the most recent kernel though), ceph luminous setup eg. no different 
kernels. 

This is 2nd or 3rd time that a vm is going into a high load (151) and 
stopping its services. I have two vm's both mounting the same 2 cephfs 
'shares'. After the last incident I dismounted the shares on the 2nd 
server. (Migrating to a new environment this 2nd server is not doing 
anything). Last time I thought maybe this could be related to my work on 
the switch from the stupid allocator to the bitmap.

Anyway yesterday I thought lets mount again the 2 shares on the 2nd 
server, see what happens. And this morning the high load was back. Afaik 
the 2nd server is only doing a cron job on the cephfs mounts, creating 
snapshots.

1) I have now still increased load on the osd nodes, from cephfs. How 
can I see what client is doing this? I don’t seem to get this from 
'ceph daemon mds.c session ls' however 'ceph osd pool stats | grep 
client -B 1' indicates it is cephfs.

2) ceph osd blacklist ls
No blacklist entries

3) the first server keeps generating such messages, while there is no 
issue with connectivity.

[Thu Jul 11 10:41:22 2019] libceph: mon0 192.168.10.111:6789 session 
lost, hunting for new mon
[Thu Jul 11 10:41:22 2019] libceph: mon1 192.168.10.112:6789 session 
established
[Thu Jul 11 10:41:22 2019] libceph: mon1 192.168.10.112:6789 io error
[Thu Jul 11 10:41:22 2019] libceph: mon1 192.168.10.112:6789 session 
lost, hunting for new mon
[Thu Jul 11 10:41:22 2019] libceph: mon0 192.168.10.111:6789 session 
established
[Thu Jul 11 10:41:22 2019] libceph: mon0 192.168.10.111:6789 io error
[Thu Jul 11 10:41:22 2019] libceph: mon0 192.168.10.111:6789 session 
lost, hunting for new mon
[Thu Jul 11 10:41:22 2019] libceph: mon2 192.168.10.113:6789 session 
established
[Thu Jul 11 10:41:22 2019] libceph: mon2 192.168.10.113:6789 io error
[Thu Jul 11 10:41:22 2019] libceph: mon2 192.168.10.113:6789 session 
lost, hunting for new mon
[Thu Jul 11 10:41:22 2019] libceph: mon0 192.168.10.111:6789 session 
established
[Thu Jul 11 10:41:22 2019] libceph: mon0 192.168.10.111:6789 io error
[Thu Jul 11 10:41:22 2019] libceph: mon0 192.168.10.111:6789 session 
lost, hunting for new mon
[Thu Jul 11 10:41:22 2019] libceph: mon2 192.168.10.113:6789 session 
established
[Thu Jul 11 10:41:22 2019] libceph: mon2 192.168.10.113:6789 io error
[Thu Jul 11 10:41:22 2019] libceph: mon2 192.168.10.113:6789 session 
lost, hunting for new mon
[Thu Jul 11 10:41:22 2019] libceph: osd25 192.168.10.114:6804 io error
[Thu Jul 11 10:41:22 2019] libceph: mon1 192.168.10.112:6789 session 
established
[Thu Jul 11 10:41:22 2019] libceph: mon1 192.168.10.112:6789 io error
[Thu Jul 11 10:41:22 2019] libceph: mon1 192.168.10.112:6789 session 
lost, hunting for new mon
[Thu Jul 11 10:41:22 2019] libceph: mon2 192.168.10.113:6789 session 
established
[Thu Jul 11 10:41:22 2019] libceph: mon2 192.168.10.113:6789 io error
[Thu Jul 11 10:41:22 2019] libceph: mon2 192.168.10.113:6789 session 
lost, hunting for new mon
[Thu Jul 11 10:41:22 2019] libceph: osd18 192.168.10.112:6802 io error
[Thu Jul 11 10:41:22 2019] libceph: mon1 192.168.10.112:6789 session 
established
[Thu Jul 11 10:41:22 2019] libceph: mon1 192.168.10.112:6789 io error
[Thu Jul 11 10:41:22 2019] libceph: mon1 192.168.10.112:6789 session 
lost, hunting for new mon
[Thu Jul 11 10:41:22 2019] libceph: mon2 192.168.10.113:6789 session 
established
[Thu Jul 11 10:41:22 2019] libceph: mon2 192.168.10.113:6789 io error
[Thu Jul 11 10:41:22 2019] libceph: mon2 192.168.10.113:6789 session 
lost, hunting for new mon
[Thu Jul 11 10:41:22 2019] libceph: osd22 192.168.10.111:6811 io error
[Thu Jul 11 10:41:22 2019] libceph: mon1 192.168.10.112:6789 session 
established
[Thu Jul 11 10:41:22 2019] libceph: mon1 192.168.10.112:6789 io error
[Thu Jul 11 10:41:22 2019] libceph: mon1 192.168.10.112:6789 session 
lost, hunting for new mon
[Thu Jul 11 10:41:22 2019] libceph: mon0 192.168.10.111:6789 session 
established

PS dmesg -T gives me strange times, as you can see these are in the 
future, os time is 2 min behind (which is the correct one, ntpd sync).
[@ ]# uptime
 10:39:17 up 50 days, 13:31,  2 users,  load average: 3.60, 3.02, 2.57

4) unmount the filesystem on the first server fails.

5) evicting the cephfs sessions of the first server, does not change the 
load of the cephfs on the osd nodes.

6) unmounting all cephfs clients, still leaves me with cephfs activity 
on the data pool and on the osd nodes.

[@c03 ~]# ceph daemon mds.c session ls
[] 

7) On the first server 
[@~]# ps -auxf| grep D
USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
root  6716  3.0  0.0  0 0 ?D10:18   0:59  \_ 
[kworker/0:2]
root 20039  0.0  0.0 123520  1212 pts/0D+   10:28   0:00  |  
 \_ umount /home/mail-archive/

[@ ~]# cat /proc/6716/stack
[] __wait_on_freeing_inode+0xb0/0xf0
[] find_inode+0x99/0xc0

Re: [ceph-users] writable snapshots in cephfs? GDPR/DSGVO

[Marc Roos]

What about creating snaps on a 'lower level' in the directory structure 
so you do not need to remove files from a snapshot as a work around?


-Original Message-
From: Lars Marowsky-Bree [mailto:l...@suse.com] 
Sent: donderdag 11 juli 2019 10:21
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] writable snapshots in cephfs? GDPR/DSGVO

On 2019-07-10T09:59:08, Lars Täuber   wrote:

> Hi everbody!
> 
> Is it possible to make snapshots in cephfs writable?
> We need to remove files because of this General Data Protection 
Regulation also from snapshots.

Removing data from existing WORM storage is tricky, snapshots being a 
specific form thereof. If you want to avoid copying and altering all 
existing records - which might clash with the requirement from other 
fields that data needs to be immutable, but I guess you could store 
checksums externally somewhere? -, this is difficult.

I think what you'd need is an additional layer - say, one holding the 
decryption keys for the tenant/user (or whatever granularity you want to 
be able to remove data at) - that you can still modify.

Once the keys have been successfully and permanently wiped, the old data 
is effectively permanently deleted (from all media; whether Ceph snaps 
or tape or other immutable storage).

You may have a record that you *had* the data.

Now, of course, you've got to manage keys, but that's significantly less 
data to massage.

Not a lawyer, either.

Good luck.


Regards,
Lars

--
SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 
21284 (AG Nürnberg) "Architects should open possibilities and not 
determine everything." (Ueli Zbinden) 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Migrating a cephfs data pool

[Marc Roos]

 
Afaik is the mv now fast because it is not moving any real data, just 
some meta data. Thus a real mv will be slow (only in the case between 
different pools) because it copies the data to the new pool and when 
successful deletes the old one. This will of course take a lot more 
time, but you at least are able to access the cephfs on both locations 
during this time and can fix things in your client access.

My problem with mv now is that if you accidentally use it between data 
pools, it does not really move data. 



-Original Message-
From: Robert LeBlanc [mailto:rob...@leblancnet.us] 
Sent: vrijdag 28 juni 2019 18:30
To: Marc Roos
Cc: ceph-users; jgarcia
Subject: Re: [ceph-users] Migrating a cephfs data pool

Given that the MDS knows everything, it seems trivial to add a ceph 'mv' 
command to do this. I looked at using tiering to try and do the move, 
but I don't know how to tell cephfs that the data is now on the new pool 
instead of the old pool name. Since we can't take a long enough downtime 
to move hundreds of Terabytes, we need something that can be done 
online, and if it has a minute or two of downtime would be okay.


Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Fri, Jun 28, 2019 at 9:02 AM Marc Roos  
wrote:


 

1.
change data pool for a folder on the file system:
setfattr -n ceph.dir.layout.pool -v fs_data.ec21 foldername

2. 
cp /oldlocation /foldername
Remember that you preferably want to use mv, but this leaves (meta) 
data 
on the old pool, that is not what you want when you want to delete 
that 
pool.

3. When everything is copied-removed, you should end up with an 
empty 
datapool with zero objects. 

4. Verify here with others, if you can just remove this one.

I think this is a reliable technique to switch, because you use the 

basic cephfs functionality that supposed to work. I prefer that the 
ceph 
guys implement a mv that does what you expect from it. Now it acts 
more 
or less like a linking.




-Original Message-
From: Jorge Garcia [mailto:jgar...@soe.ucsc.edu] 
Sent: vrijdag 28 juni 2019 17:52
To: Marc Roos; ceph-users
Subject: Re: [ceph-users] Migrating a cephfs data pool

Are you talking about adding the new data pool to the current 
filesystem? Like:

   $ ceph fs add_data_pool my_ceph_fs new_ec_pool

I have done that, and now the filesystem shows up as having two 
data 
pools:

   $ ceph fs ls
   name: my_ceph_fs, metadata pool: cephfs_meta, data pools: 
[cephfs_data new_ec_pool ]

but then I run into two issues:

1. How do I actually copy/move/migrate the data from the old pool 
to the 
new pool?
2. When I'm done moving the data, how do I get rid of the old data 
pool? 

I know there's a rm_data_pool option, but I have read on the 
mailing 
list that you can't remove the original data pool from a cephfs 
filesystem.

The other option is to create a whole new cephfs with a new 
metadata 
pool and the new data pool, but creating multiple filesystems is 
still 
experimental and not allowed by default...

On 6/28/19 8:28 AM, Marc Roos wrote:
>   
> What about adding the new data pool, mounting it and then moving 
the 
> files? (read copy because move between data pools does not what 
you 
> expect it do)
>
>
> -Original Message-
> From: Jorge Garcia [mailto:jgar...@soe.ucsc.edu]
> Sent: vrijdag 28 juni 2019 17:26
> To: ceph-users
> Subject: *SPAM* [ceph-users] Migrating a cephfs data pool
>
> This seems to be an issue that gets brought up repeatedly, but I 
> haven't seen a definitive answer yet. So, at the risk of 
repeating a 
> question that has already been asked:
>
> How do you migrate a cephfs data pool to a new data pool? The 
obvious 
> case would be somebody that has set up a replicated pool for 
their 
> cephfs data and then wants to convert it to an erasure code pool. 
Is 
> there a simple way to do this, other than creating a whole new 
ceph 
> cluster and copying the data using rsync?
>
> Thanks for any clues
>
> Jorge
>
> ___
> ceph-users mailing list
> cep

Re: [ceph-users] Migrating a cephfs data pool

[Marc Roos]

 

1.
change data pool for a folder on the file system:
setfattr -n ceph.dir.layout.pool -v fs_data.ec21 foldername

2. 
cp /oldlocation /foldername
Remember that you preferably want to use mv, but this leaves (meta) data 
on the old pool, that is not what you want when you want to delete that 
pool.

3. When everything is copied-removed, you should end up with an empty 
datapool with zero objects. 

4. Verify here with others, if you can just remove this one.

I think this is a reliable technique to switch, because you use the 
basic cephfs functionality that supposed to work. I prefer that the ceph 
guys implement a mv that does what you expect from it. Now it acts more 
or less like a linking.




-Original Message-
From: Jorge Garcia [mailto:jgar...@soe.ucsc.edu] 
Sent: vrijdag 28 juni 2019 17:52
To: Marc Roos; ceph-users
Subject: Re: [ceph-users] Migrating a cephfs data pool

Are you talking about adding the new data pool to the current 
filesystem? Like:

   $ ceph fs add_data_pool my_ceph_fs new_ec_pool

I have done that, and now the filesystem shows up as having two data 
pools:

   $ ceph fs ls
   name: my_ceph_fs, metadata pool: cephfs_meta, data pools: 
[cephfs_data new_ec_pool ]

but then I run into two issues:

1. How do I actually copy/move/migrate the data from the old pool to the 
new pool?
2. When I'm done moving the data, how do I get rid of the old data pool? 

I know there's a rm_data_pool option, but I have read on the mailing 
list that you can't remove the original data pool from a cephfs 
filesystem.

The other option is to create a whole new cephfs with a new metadata 
pool and the new data pool, but creating multiple filesystems is still 
experimental and not allowed by default...

On 6/28/19 8:28 AM, Marc Roos wrote:
>   
> What about adding the new data pool, mounting it and then moving the 
> files? (read copy because move between data pools does not what you 
> expect it do)
>
>
> -Original Message-
> From: Jorge Garcia [mailto:jgar...@soe.ucsc.edu]
> Sent: vrijdag 28 juni 2019 17:26
> To: ceph-users
> Subject: *SPAM* [ceph-users] Migrating a cephfs data pool
>
> This seems to be an issue that gets brought up repeatedly, but I 
> haven't seen a definitive answer yet. So, at the risk of repeating a 
> question that has already been asked:
>
> How do you migrate a cephfs data pool to a new data pool? The obvious 
> case would be somebody that has set up a replicated pool for their 
> cephfs data and then wants to convert it to an erasure code pool. Is 
> there a simple way to do this, other than creating a whole new ceph 
> cluster and copying the data using rsync?
>
> Thanks for any clues
>
> Jorge
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Migrating a cephfs data pool

[Marc Roos]

 
What about adding the new data pool, mounting it and then moving the 
files? (read copy because move between data pools does not what you 
expect it do)


-Original Message-
From: Jorge Garcia [mailto:jgar...@soe.ucsc.edu] 
Sent: vrijdag 28 juni 2019 17:26
To: ceph-users
Subject: *SPAM* [ceph-users] Migrating a cephfs data pool

This seems to be an issue that gets brought up repeatedly, but I haven't 
seen a definitive answer yet. So, at the risk of repeating a question 
that has already been asked:

How do you migrate a cephfs data pool to a new data pool? The obvious 
case would be somebody that has set up a replicated pool for their 
cephfs data and then wants to convert it to an erasure code pool. Is 
there a simple way to do this, other than creating a whole new ceph 
cluster and copying the data using rsync?

Thanks for any clues

Jorge

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Expected IO in luminous Ceph Cluster

[Marc Roos]

I am also thinking of moving the wal/db to ssd of the sata hdd's. Did 
you do tests before and after this change, and know what the difference 
is iops? And is the advantage more or less when your sata hdd's are 
slower? 


-Original Message-
From: Stolte, Felix [mailto:f.sto...@fz-juelich.de] 
Sent: donderdag 6 juni 2019 10:47
To: ceph-users
Subject: [ceph-users] Expected IO in luminous Ceph Cluster

Hello folks,

we are running a ceph cluster on Luminous consisting of 21 OSD Nodes 
with 9 8TB SATA drives and 3 Intel 3700 SSDs for Bluestore WAL and DB 
(1:3 Ratio). OSDs have 10Gb for Public and Cluster Network. The cluster 
is running stable for over a year. We didn’t had a closer look on IO 
until one of our customers started to complain about a VM we migrated 
from VMware with Netapp Storage to our Openstack Cloud with ceph 
storage. He sent us a sysbench report from the machine, which I could 
reproduce on other VMs as well as on a mounted RBD on physical hardware:

sysbench --file-fsync-freq=1 --threads=16 fileio --file-total-size=1G 
--file-test-mode=rndrw --file-rw-ratio=2 run sysbench 1.0.11 (using 
system LuaJIT 2.1.0-beta3)

Running the test with following options:
Number of threads: 16
Initializing random number generator from current time

Extra file open flags: 0
128 files, 8MiB each
1GiB total file size
Block size 16KiB
Number of IO requests: 0
Read/Write ratio for combined random IO test: 2.00 Periodic FSYNC 
enabled, calling fsync() each 1 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test

File operations:
reads/s:  36.36
writes/s: 18.18
fsyncs/s: 2318.59

Throughput:
read, MiB/s:  0.57
written, MiB/s:   0.28

General statistics:
total time:  10.0071s
total number of events:  23755

Latency (ms):
 min:  0.01
 avg:  6.74
 max:   1112.58
 95th percentile: 26.68
 sum: 160022.67

Threads fairness:
events (avg/stddev):   1484.6875/52.59
execution time (avg/stddev):   10.0014/0.00

Are these numbers reasonable for a cluster of our size?

Best regards
Felix
IT-Services
Telefon 02461 61-9243
E-Mail: f.sto...@fz-juelich.de

-

-
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), 
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. 
Dr. Sebastian M. Schmidt

-

-
 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Single threaded IOPS on SSD pool.

[Marc Roos]

 We have this, if it is any help

write-4k-seq: (groupid=0, jobs=1): err= 0: pid=1446964: Fri May 24 
19:41:48 2019
  write: IOPS=760, BW=3042KiB/s (3115kB/s)(535MiB/180001msec)
slat (usec): min=7, max=234, avg=16.59, stdev=13.59
clat (usec): min=786, max=167483, avg=1295.60, stdev=1933.25
 lat (usec): min=810, max=167492, avg=1312.46, stdev=1933.67
clat percentiles (usec):
 |  1.00th=[   914],  5.00th=[   979], 10.00th=[  1020], 20.00th=[  
1074],
 | 30.00th=[  1123], 40.00th=[  1172], 50.00th=[  1205], 60.00th=[  
1254],
 | 70.00th=[  1319], 80.00th=[  1401], 90.00th=[  1516], 95.00th=[  
1631],
 | 99.00th=[  2704], 99.50th=[  3949], 99.90th=[  5145], 99.95th=[  
5538],
 | 99.99th=[139461]
   bw (  KiB/s): min=  625, max= 3759, per=80.13%, avg=2436.63, 
stdev=653.68, samples=359
   iops: min=  156, max=  939, avg=608.76, stdev=163.41, 
samples=359
  lat (usec)   : 1000=7.83%
  lat (msec)   : 2=90.27%, 4=1.42%, 10=0.45%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.02%
  cpu  : usr=0.52%, sys=1.55%, ctx=162087, majf=0, minf=28
  IO depths: 1=117.6%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 
>=64=0.0%
 submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0%
 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0%
 issued rwt: total=0,136889,0, short=0,0,0, dropped=0,0,0
 latency   : target=0, window=0, percentile=100.00%, depth=1
randwrite-4k-seq: (groupid=1, jobs=1): err= 0: pid=1448032: Fri May 24 
19:41:48 2019
  write: IOPS=655, BW=2620KiB/s (2683kB/s)(461MiB/180001msec)
slat (usec): min=7, max=120, avg=10.79, stdev= 6.22
clat (usec): min=897, max=77251, avg=1512.76, stdev=368.36
 lat (usec): min=906, max=77262, avg=1523.77, stdev=368.54
clat percentiles (usec):
 |  1.00th=[ 1106],  5.00th=[ 1205], 10.00th=[ 1254], 20.00th=[ 
1319],
 | 30.00th=[ 1369], 40.00th=[ 1418], 50.00th=[ 1483], 60.00th=[ 
1532],
 | 70.00th=[ 1598], 80.00th=[ 1663], 90.00th=[ 1778], 95.00th=[ 
1893],
 | 99.00th=[ 2540], 99.50th=[ 2933], 99.90th=[ 3392], 99.95th=[ 
4080],
 | 99.99th=[ 6194]
   bw (  KiB/s): min= 1543, max= 2830, per=79.66%, avg=2087.02, 
stdev=396.14, samples=359
   iops: min=  385, max=  707, avg=521.39, stdev=99.06, 
samples=359
  lat (usec)   : 1000=0.06%
  lat (msec)   : 2=97.19%, 4=2.70%, 10=0.04%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu  : usr=0.39%, sys=1.13%, ctx=118477, majf=0, minf=50
  IO depths: 1=116.6%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 
>=64=0.0%
 submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0%
 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0%
 issued rwt: total=0,117905,0, short=0,0,0, dropped=0,0,0
 latency   : target=0, window=0, percentile=100.00%, depth=1
read-4k-seq: (groupid=2, jobs=1): err= 0: pid=1449103: Fri May 24 
19:41:48 2019
   read: IOPS=2736, BW=10.7MiB/s (11.2MB/s)(1924MiB/180001msec)
slat (usec): min=6, max=142, avg= 9.26, stdev= 5.02
clat (usec): min=152, max=13253, avg=353.73, stdev=98.92
 lat (usec): min=160, max=13262, avg=363.24, stdev=99.15
clat percentiles (usec):
 |  1.00th=[  182],  5.00th=[  215], 10.00th=[  239], 20.00th=[  
273],
 | 30.00th=[  306], 40.00th=[  330], 50.00th=[  355], 60.00th=[  
375],
 | 70.00th=[  396], 80.00th=[  420], 90.00th=[  461], 95.00th=[  
498],
 | 99.00th=[  586], 99.50th=[  635], 99.90th=[  775], 99.95th=[  
889],
 | 99.99th=[ 1958]
   bw (  KiB/s): min= 5883, max=13817, per=79.66%, avg=8720.01, 
stdev=1895.05, samples=359
   iops: min= 1470, max= 3454, avg=2179.63, stdev=473.78, 
samples=359
  lat (usec)   : 250=13.13%, 500=82.11%, 750=4.64%, 1000=0.09%
  lat (msec)   : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu  : usr=1.31%, sys=3.69%, ctx=493433, majf=0, minf=32
  IO depths: 1=115.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 
>=64=0.0%
 submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0%
 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0%
 issued rwt: total=492640,0,0, short=0,0,0, dropped=0,0,0
 latency   : target=0, window=0, percentile=100.00%, depth=1
randread-4k-seq: (groupid=3, jobs=1): err= 0: pid=1450173: Fri May 24 
19:41:48 2019
   read: IOPS=1812, BW=7251KiB/s (7425kB/s)(1275MiB/180001msec)
slat (usec): min=6, max=161, avg=10.25, stdev= 6.37
clat (usec): min=182, max=23748, avg=538.35, stdev=136.71
 lat (usec): min=189, max=23758, avg=548.86, stdev=137.19
clat percentiles (usec):
 |  1.00th=[  265],  5.00th=[  310], 10.00th=[  351], 20.00th=[  
445],
 | 30.00th=[  494], 40.00th=[  519], 50.00th=[  537], 60.00th=[  
562],
 | 70.00th=[  594], 80.00th=[  644], 90.00th=[  701], 95.00th=[  
742],
 | 99.00th=[  816], 99.50th=[  840], 99.90th=[  914], 99.95th=[ 
1172],
 | 99.99th=[ 2442]
   bw (  KiB/s): min= 4643, 

Re: [ceph-users] How to remove ceph-mgr from a node

[Marc Roos]

 
What is wrong with?

service ceph-mgr@c stop
systemctl disable ceph-mgr@c


-Original Message-
From: Vandeir Eduardo [mailto:vandeir.edua...@gmail.com] 
Sent: woensdag 5 juni 2019 16:44
To: ceph-users
Subject: [ceph-users] How to remove ceph-mgr from a node

Hi guys,

sorry, but I'm not finding in documentation how to remove ceph-mgr from 
a node. Is it possible?

Thanks.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Radosgw in container

[Marc Roos]

Has anyone put the radosgw in a container? What files do I need to put 
in the sandbox directory? Are there other things I should consider?



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CEPH MDS Damaged Metadata - recovery steps

[Marc Roos]

 
How did this get damaged? You had 3x replication on the pool?



-Original Message-
From: Yan, Zheng [mailto:uker...@gmail.com] 
Sent: dinsdag 4 juni 2019 1:14
To: James Wilkins
Cc: ceph-users
Subject: Re: [ceph-users] CEPH MDS Damaged Metadata - recovery steps

On Mon, Jun 3, 2019 at 3:06 PM James Wilkins 
 wrote:
>
> Hi all,
>
> After a bit of advice to ensure we’re approaching this the right way.
>
> (version: 12.2.12, multi-mds, dirfrag is enabled)
>
> We have corrupt meta-data as identified by ceph
>
> health: HEALTH_ERR
> 2 MDSs report damaged metadata
>
> Asking the mds via damage ls
>
> {
> "damage_type": "dir_frag",
> "id": 2265410500,
> "ino": 2199349051809,
> "frag": "*",
> "path": 
"/projects/17343-5bcdaf07f4055-managed-server-0/apache-echfq-data/html/s
hop/app/cache/prod/smarty/cache/iqitreviews/simple/21832/1"
> }
>
>
> We’ve done the steps outlined here -> 
> http://docs.ceph.com/docs/luminous/cephfs/disaster-recovery/ namely
>
> cephfs-journal-tool –fs:all journal reset (both ranks) 
> cephfs-data-scan scan extents / inodes / links has completed
>
> However when attempting to access the named folder we get:
>
> 2019-05-31 03:16:04.792274 7f56f6fb5700 -1 log_channel(cluster) log 
> [ERR] : dir 0x200136b41a1 object missing on disk; some files may be 
> lost 
> (/projects/17343-5bcdaf07f4055-managed-server-0/apache-echfq-data/html
> /shop/app/cache/prod/smarty/cache/iqitreviews/simple/21832/1)
>
> We get this error followed shortly by an MDS failover
>
> Two questions really
>
> What’s not immediately clear from the documentation is should we/do 
we also need to run the below?
>
> # Session table
> cephfs-table-tool 0 reset session
> # SnapServer
> cephfs-table-tool 0 reset snap
> # InoTable
> cephfs-table-tool 0 reset inode
> # Root inodes ("/" and MDS directory)
> cephfs-data-scan init
>

No, don't do this.

> And secondly – our current train of thought is we need to grab the 
inode number of the parent folder and delete this from the metadata pool 
via rados rmomapkey – is this correct?
>

Yes, find inode number of directory 21832. check if omap key '1_head'
exist in object .. If it exists, 
remove it.

> Any input appreciated
>
> Cheers,
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Balancer: uneven OSDs

[Marc Roos]

 

I had this with balancer active and "crush-compat"
MIN/MAX VAR: 0.43/1.59  STDDEV: 10.81

And by increasing the pg of some pools (from 8 to 64) and deleting empty 
pools, I went to this

MIN/MAX VAR: 0.59/1.28  STDDEV: 6.83

(Do not want to go to this upmap yet)




-Original Message-
From: Tarek Zegar [mailto:tze...@us.ibm.com] 
Sent: woensdag 29 mei 2019 17:52
To: ceph-users
Subject: *SPAM* [ceph-users] Balancer: uneven OSDs

Can anyone help with this? Why can't I optimize this cluster, the pg 
counts and data distribution is way off.
__

I enabled the balancer plugin and even tried to manually invoke it but 
it won't allow any changes. Looking at ceph osd df, it's not even at 
all. Thoughts?

root@hostadmin:~# ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
1 hdd 0.00980 0 0 B 0 B 0 B 0 0 0
3 hdd 0.00980 1.0 10 GiB 8.3 GiB 1.7 GiB 82.83 1.14 156
6 hdd 0.00980 1.0 10 GiB 8.4 GiB 1.6 GiB 83.77 1.15 144 0 hdd 
0.00980 0 0 B 0 B 0 B 0 0 0
5 hdd 0.00980 1.0 10 GiB 9.0 GiB 1021 MiB 90.03 1.23 159
7 hdd 0.00980 1.0 10 GiB 7.7 GiB 2.3 GiB 76.57 1.05 141
2 hdd 0.00980 1.0 10 GiB 5.5 GiB 4.5 GiB 55.42 0.76 90
4 hdd 0.00980 1.0 10 GiB 5.9 GiB 4.1 GiB 58.78 0.81 99
8 hdd 0.00980 1.0 10 GiB 6.3 GiB 3.7 GiB 63.12 0.87 111 TOTAL 90 GiB 
53 GiB 37 GiB 72.93 MIN/MAX VAR: 0.76/1.23 STDDEV: 12.67


root@hostadmin:~# osdmaptool om --upmap out.txt --upmap-pool rbd
osdmaptool: osdmap file 'om'
writing upmap command output to: out.txt checking for upmap cleanups 
upmap, max-count 100, max deviation 0.01 <---really? It's not even close 
to 1% across the drives limiting to pools rbd (1) no upmaps proposed


ceph balancer optimize myplan
Error EALREADY: Unable to find further optimization,or distribution is 
already perfect



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic

[Marc Roos]

I switched first of may, and did not notice to much difference in memory 
usage. After the restart of the osd's on the node I see the memory 
consumption gradually getting back to as before.
Can't say anything about latency.






-Original Message-
From: Konstantin Shalygin  
Sent: dinsdag 28 mei 2019 11:52
To: Wido den Hollander
Cc: ceph-users@lists.ceph.com
Subject: *SPAM* Re: [ceph-users] BlueStore bitmap allocator 
under Luminous and Mimic



Hi,

With the release of 12.2.12 the bitmap allocator for BlueStore is 
now
available under Mimic and Luminous.

[osd]
bluestore_allocator = bitmap
bluefs_allocator = bitmap

Before setting this in production: What might the implications be 
and
what should be thought of?

>From what I've read the bitmap allocator seems to be better in 
read
performance and uses less memory.

In Nautilus bitmap is the default, but L and M still default to 
stupid.

Since the bitmap allocator was backported there must be a use-case 
to
use the bitmap allocator instead of stupid.

Thanks!

Wido




Wido, do you setted allocator to bitmap on L installations past this 
months? Any improvements?







k





test-memory.png
Description: Binary data
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] performance in a small cluster

[Marc Roos]

 
Maybe my data can be useful to compare with? I have the samsung sm863. 

This[0] is what I get from fio directly on the ssd, and from an rbd ssd 
pool with 3x replication[1]. 
I also have included a comparisson with cephfs[3], would be nice if 
there would be some sort of
 manual page describing general to be expected ceph overhead.


[0] direct
randwrite-4k-seq: (groupid=1, jobs=1): err= 0: pid=522903: Thu Sep  6 
21:04:12 2018
  write: IOPS=17.9k, BW=69.8MiB/s (73.2MB/s)(12.3GiB/180001msec)
slat (usec): min=4, max=333, avg= 9.94, stdev= 5.00
clat (nsec): min=1141, max=1131.2k, avg=42560.69, stdev=9074.14
 lat (usec): min=35, max=1137, avg=52.80, stdev= 9.42
clat percentiles (usec):
 |  1.00th=[   33],  5.00th=[   35], 10.00th=[   35], 20.00th=[   
35],
 | 30.00th=[   36], 40.00th=[   36], 50.00th=[   41], 60.00th=[   
43],
 | 70.00th=[   49], 80.00th=[   54], 90.00th=[   57], 95.00th=[   
58],
 | 99.00th=[   60], 99.50th=[   62], 99.90th=[   67], 99.95th=[   
70],
 | 99.99th=[  174]
   bw (  KiB/s): min=34338, max=92268, per=84.26%, avg=60268.13, 
stdev=12283.36, samples=359
   iops: min= 8584, max=23067, avg=15066.67, stdev=3070.87, 
samples=359
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=71.73%, 100=28.24%
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%
  cpu  : usr=12.96%, sys=26.87%, ctx=3218988, majf=0, minf=10962
  IO depths: 1=116.8%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 
>=64=0.0%
 submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0%
 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0%
 issued rwt: total=0,3218724,0, short=0,0,0, dropped=0,0,0
 latency   : target=0, window=0, percentile=100.00%, depth=1
randread-4k-seq: (groupid=3, jobs=1): err= 0: pid=523297: Thu Sep  6 
21:04:12 2018
   read: IOPS=10.2k, BW=39.7MiB/s (41.6MB/s)(7146MiB/180001msec)
slat (usec): min=4, max=328, avg=15.39, stdev= 8.62
clat (nsec): min=1600, max=948792, avg=78946.53, stdev=36246.91
 lat (usec): min=39, max=969, avg=94.75, stdev=37.43
clat percentiles (usec):
 |  1.00th=[   38],  5.00th=[   40], 10.00th=[   40], 20.00th=[   
41],
 | 30.00th=[   41], 40.00th=[   52], 50.00th=[   70], 60.00th=[  
110],
 | 70.00th=[  112], 80.00th=[  115], 90.00th=[  125], 95.00th=[  
127],
 | 99.00th=[  133], 99.50th=[  135], 99.90th=[  141], 99.95th=[  
147],
 | 99.99th=[  243]
   bw (  KiB/s): min=19918, max=49336, per=84.40%, avg=34308.52, 
stdev=6891.67, samples=359
   iops: min= 4979, max=12334, avg=8576.75, stdev=1722.92, 
samples=359
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=38.06%, 100=19.88%
  lat (usec)   : 250=42.04%, 500=0.01%, 750=0.01%, 1000=0.01%
  cpu  : usr=8.07%, sys=21.59%, ctx=1829588, majf=0, minf=10954
  IO depths: 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 
>=64=0.0%
 submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0%
 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0%
 issued rwt: total=1829296,0,0, short=0,0,0, dropped=0,0,0
 latency   : target=0, window=0, percentile=100.00%, depth=1

[1] rbd ssd 3x
randwrite-4k-seq: (groupid=1, jobs=1): err= 0: pid=1448032: Fri May 24 
19:41:48 2019
  write: IOPS=655, BW=2620KiB/s (2683kB/s)(461MiB/180001msec)
slat (usec): min=7, max=120, avg=10.79, stdev= 6.22
clat (usec): min=897, max=77251, avg=1512.76, stdev=368.36
 lat (usec): min=906, max=77262, avg=1523.77, stdev=368.54
clat percentiles (usec):
 |  1.00th=[ 1106],  5.00th=[ 1205], 10.00th=[ 1254], 20.00th=[ 
1319],
 | 30.00th=[ 1369], 40.00th=[ 1418], 50.00th=[ 1483], 60.00th=[ 
1532],
 | 70.00th=[ 1598], 80.00th=[ 1663], 90.00th=[ 1778], 95.00th=[ 
1893],
 | 99.00th=[ 2540], 99.50th=[ 2933], 99.90th=[ 3392], 99.95th=[ 
4080],
 | 99.99th=[ 6194]
   bw (  KiB/s): min= 1543, max= 2830, per=79.66%, avg=2087.02, 
stdev=396.14, samples=359
   iops: min=  385, max=  707, avg=521.39, stdev=99.06, 
samples=359
  lat (usec)   : 1000=0.06%
  lat (msec)   : 2=97.19%, 4=2.70%, 10=0.04%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu  : usr=0.39%, sys=1.13%, ctx=118477, majf=0, minf=50
  IO depths: 1=116.6%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 
>=64=0.0%
 submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0%
 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0%
 issued rwt: total=0,117905,0, short=0,0,0, dropped=0,0,0
 latency   : target=0, window=0, percentile=100.00%, depth=1
randread-4k-seq: (groupid=3, jobs=1): err= 0: pid=1450173: Fri May 24 
19:41:48 2019
   read: IOPS=1812, BW=7251KiB/s (7425kB/s)(1275MiB/180001msec)
slat (usec): min=6, max=161, avg=10.25, stdev= 6.37
clat (usec): min=182, max=23748, avg=538.35, stdev=136.71
 lat (usec): min=189, max=23758, avg=548.86, stdev=137.19
clat 

[ceph-users] "allow profile rbd" or "profile rbd"

[Marc Roos]

I have still some account listing either "allow" or not. What should 
this be? Should this not be kept uniform?



[client.xxx.xx]
 key = xxx
 caps mon = "allow profile rbd"
 caps osd = "profile rbd pool=rbd,profile rbd pool=rbd.ssd"



[client.xxx]
 key = 
 caps mon = "profile rbd"





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph dovecot

[Marc Roos]

Sorry for not waiting until it is published on the ceph website but, 
anyone attended this talk? Is it production ready? 

https://cephalocon2019.sched.com/event/M7j8
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Major ceph disaster

[Marc Roos]

I have been following this thread for a while, and thought I need to 
have 
 "major ceph disaster" alert on the monitoring ;)
 http://www.f1-outsourcing.eu/files/ceph-disaster.mp4 




-Original Message-
From: Kevin Flöh [mailto:kevin.fl...@kit.edu] 
Sent: donderdag 23 mei 2019 10:51
To: Robert LeBlanc
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Major ceph disaster

Hi,

we have set the PGs to recover and now they are stuck in 
active+recovery_wait+degraded and instructing them to deep-scrub does 
not change anything. Hence, the rados report is empty. Is there a way to 
stop the recovery wait to start the deep-scrub and get the output? I 
guess the recovery_wait might be caused by missing objects. Do we need 
to delete them first to get the recovery going?


Kevin


On 22.05.19 6:03 nachm., Robert LeBlanc wrote:


On Wed, May 22, 2019 at 4:31 AM Kevin Flöh  
wrote:


Hi,

thank you, it worked. The PGs are not incomplete anymore. 
Still we have 
another problem, there are 7 PGs inconsistent and a cpeh pg 
repair is 
not doing anything. I just get "instructing pg 1.5dd on osd.24 
to 
repair" and nothing happens. Does somebody know how we can get 
the PGs 
to repair?

Regards,

Kevin



Kevin,

I just fixed an inconsistent PG yesterday. You will need to figure 
out why they are inconsistent. Do these steps and then we can figure out 
how to proceed.
1. Do a deep-scrub on each PG that is inconsistent. (This may fix 
some of them)
2. Print out the inconsistent report for each inconsistent PG. 
`rados list-inconsistent-obj  --format=json-pretty`
3. You will want to look at the error messages and see if all the 
shards have the same data.

Robert LeBlanc
 


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to fix this? session lost, hunting for new mon, session established, io error

[Marc Roos]

I am still stuck with this situation, and do not want to restart(reset) 
this host. I tried bringing down the eth connected to the client network 
for a while, but after bringing it up, I am getting the same messages 



-Original Message-
From: Marc Roos 
Sent: dinsdag 21 mei 2019 11:42
To: ceph-users
Subject: [ceph-users] How to fix this? session lost, hunting for new 
mon, session established, io error



I have this on a cephfs client, I had ceph common on 12.2.11, and 
upgraded to 12.2.12 while having this error. They are writing here [0] 
you need to upgrade kernel and it is fixed in 12.2.2

[@~]# uname -a
Linux mail03 3.10.0-957.5.1.el7.x86_6

[Tue May 21 11:23:26 2019] libceph: mon2 192.168.10.113:6789 session 
established
[Tue May 21 11:23:26 2019] libceph: mon2 192.168.10.113:6789 io error
[Tue May 21 11:23:26 2019] libceph: mon2 192.168.10.113:6789 session 
lost, hunting for new mon
[Tue May 21 11:23:26 2019] libceph: mon0 192.168.10.111:6789 session 
established
[Tue May 21 11:23:26 2019] libceph: mon0 192.168.10.111:6789 io error
[Tue May 21 11:23:26 2019] libceph: mon0 192.168.10.111:6789 session 
lost, hunting for new mon
[Tue May 21 11:23:26 2019] libceph: mon1 192.168.10.112:6789 session 
established
[Tue May 21 11:23:26 2019] libceph: mon1 192.168.10.112:6789 
[Tue May 21 11:23:26 2019] libceph: mon1 192.168.10.112:6789 session 
lost, hunting for new mon
[Tue May 21 11:23:26 2019] libceph: mon2 192.168.10.113:6789 session 
established



ceph version
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous 

(stable)

[0]
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg52177.html
https://tracker.ceph.com/issues/23537
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] How to fix this? session lost, hunting for new mon, session established, io error

[Marc Roos]

I have this on a cephfs client, I had ceph common on 12.2.11, and 
upgraded to 12.2.12 while having this error. They are writing here [0] 
you need to upgrade kernel and it is fixed in 12.2.2

[@~]# uname -a
Linux mail03 3.10.0-957.5.1.el7.x86_6

[Tue May 21 11:23:26 2019] libceph: mon2 192.168.10.113:6789 session 
established
[Tue May 21 11:23:26 2019] libceph: mon2 192.168.10.113:6789 io error
[Tue May 21 11:23:26 2019] libceph: mon2 192.168.10.113:6789 session 
lost, hunting for new mon
[Tue May 21 11:23:26 2019] libceph: mon0 192.168.10.111:6789 session 
established
[Tue May 21 11:23:26 2019] libceph: mon0 192.168.10.111:6789 io error
[Tue May 21 11:23:26 2019] libceph: mon0 192.168.10.111:6789 session 
lost, hunting for new mon
[Tue May 21 11:23:26 2019] libceph: mon1 192.168.10.112:6789 session 
established
[Tue May 21 11:23:26 2019] libceph: mon1 192.168.10.112:6789 
[Tue May 21 11:23:26 2019] libceph: mon1 192.168.10.112:6789 session 
lost, hunting for new mon
[Tue May 21 11:23:26 2019] libceph: mon2 192.168.10.113:6789 session 
established



ceph version
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous 
(stable)

[0]
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg52177.html
https://tracker.ceph.com/issues/23537
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Cephfs client evicted, how to unmount the filesystem on the client?

[Marc Roos]

[@ceph]# ps -aux | grep D
USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
root 12527  0.0  0.0 123520   932 pts/1D+   09:26   0:00 umount 
/home/mail-archive
root 14549  0.2  0.0  0 0 ?D09:29   0:09 
[kworker/0:0]
root 23350  0.0  0.0 123520   932 pts/0D09:38   0:00 umount 
/home/archiveindex

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Is a not active mds doing something?

[Marc Roos]

I have not configured anything for the msd except this


[mds]
# 100k+ files in 2 folders
mds bal fragment size max = 12
# maybe for nfs-ganesha problems?
# http://docs.ceph.com/docs/master/cephfs/eviction/
mds_session_blacklist_on_timeout = false
mds_session_blacklist_on_evict = false
mds_cache_memory_limit = 80
# faster fail over?
#mds beacon grace = 5


-Original Message-
From: Eugen Block [mailto:ebl...@nde.ag] 
Sent: dinsdag 21 mei 2019 10:18
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Is a not active mds doing something?

Hi Marc,

have you configured the other MDS to be standby-replay for the active 
MDS? I have three MDS servers, one is active, the second is 
active-standby and the third just standby. If the active fails, the 
second takes over within seconds. This is what I have in my ceph.conf:

[mds.]
mds_standby_replay = true
mds_standby_for_rank = 0


Regards,
Eugen


Zitat von Marc Roos :

> Should a not active mds be doing something??? When I restarted the not 

> active mds.c, My client io on the fs_data pool disappeared.
>
>
>   services:
> mon: 3 daemons, quorum a,b,c
> mgr: c(active), standbys: a, b
> mds: cephfs-1/1/1 up  {0=a=up:active}, 1 up:standby
> osd: 32 osds: 32 up, 32 in
> rgw: 2 daemons active
>
>
>
> -----Original Message-
> From: Marc Roos
> Sent: dinsdag 21 mei 2019 10:01
> To: ceph-users@lists.ceph.com; Marc Roos
> Subject: RE: [ceph-users] cephfs causing high load on vm, taking down 
> 15 min later another cephfs vm
>
>
> I have evicted all client connections and have still high load on 
> osd's
>
> And ceph osd pool stats shows still client activity?
>
> pool fs_data id 20
>   client io 565KiB/s rd, 120op/s rd, 0op/s wr
>
>
>
>
> -Original Message-
> From: Marc Roos
> Sent: dinsdag 21 mei 2019 9:51
> To: ceph-users@lists.ceph.com; Marc Roos
> Subject: RE: [ceph-users] cephfs causing high load on vm, taking down 
> 15 min later another cephfs vm
>
>
> I have got this today again? I cannot unmount the filesystem and 
> looks like some osd's are having 100% cpu utilization?
>
>
> -Original Message-
> From: Marc Roos
> Sent: maandag 20 mei 2019 12:42
> To: ceph-users
> Subject: [ceph-users] cephfs causing high load on vm, taking down 15 
> min later another cephfs vm
>
>
>
> I got my first problem with cephfs in a production environment. Is it 
> possible from these logfiles to deduct what happened?
>
> svr1 is connected to ceph client network via switch
> svr2 vm is collocated on c01 node.
> c01 has osd's and the mon.a colocated.
>
> svr1 was the first to report errors at 03:38:44. I have no error 
> messages reported of a network connection problem by any of the ceph 
> nodes. I have nothing in dmesg on c01.
>
> [@c01 ~]# cat /etc/redhat-release
> CentOS Linux release 7.6.1810 (Core)
> [@c01 ~]# uname -a
> Linux c01 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 
> 2019
>
> x86_64 x86_64 x86_64 GNU/Linux
> [@c01 ~]# ceph versions
> {
> "mon": {
> "ceph version 12.2.12 
> (1436006594665279fe734b4c15d7e08c13ebd777)
>
> luminous (stable)": 3
> },
> "mgr": {
> "ceph version 12.2.12 
> (1436006594665279fe734b4c15d7e08c13ebd777)
>
> luminous (stable)": 3
> },
> "osd": {
> "ceph version 12.2.12 
> (1436006594665279fe734b4c15d7e08c13ebd777)
>
> luminous (stable)": 32
> },
> "mds": {
> "ceph version 12.2.12 
> (1436006594665279fe734b4c15d7e08c13ebd777)
>
> luminous (stable)": 2
> },
> "rgw": {
> "ceph version 12.2.12 
> (1436006594665279fe734b4c15d7e08c13ebd777)
>
> luminous (stable)": 2
> },
> "overall": {
> "ceph version 12.2.12 
> (1436006594665279fe734b4c15d7e08c13ebd777)
>
> luminous (stable)": 42
> }
> }
>
>
>
>
> [0] svr1 messages
> May 20 03:36:01 svr1 systemd: Started Session 308978 of user root.
> May 20 03:36:01 svr1 systemd: Started Session 308979 of user root.
> May 20 03:36:01 svr1 systemd: Started Session 308979 of user root.
> May 20 03:36:01 svr1 systemd: Started Session 308980 of user root.
> May 20 03:36:01 svr1 systemd: Started Session 308980 of user root.
> May 20 03:38:01 svr1 systemd: Started Session 308981 of user root.
> May 20 03:38:01 svr1 systemd: Started Session 308981 of user root.
> May 20 03:38:01 svr1 systemd: Started Session 308982 of user root.
> May 20 03:38:01 svr1 systemd: Started Session 308982 of user root.
> May 20 03:38:01 svr1 systemd: Started Session 308983 of 

[ceph-users] Is a not active mds doing something?

[Marc Roos]

Should a not active mds be doing something??? When I restarted the not 
active mds.c, My client io on the fs_data pool disappeared. 


  services:
mon: 3 daemons, quorum a,b,c
mgr: c(active), standbys: a, b
mds: cephfs-1/1/1 up  {0=a=up:active}, 1 up:standby
osd: 32 osds: 32 up, 32 in
rgw: 2 daemons active



-Original Message-
From: Marc Roos 
Sent: dinsdag 21 mei 2019 10:01
To: ceph-users@lists.ceph.com; Marc Roos
Subject: RE: [ceph-users] cephfs causing high load on vm, taking down 15 
min later another cephfs vm

 
I have evicted all client connections and have still high load on osd's 

And ceph osd pool stats shows still client activity?

pool fs_data id 20
  client io 565KiB/s rd, 120op/s rd, 0op/s wr




-Original Message-
From: Marc Roos
Sent: dinsdag 21 mei 2019 9:51
To: ceph-users@lists.ceph.com; Marc Roos
Subject: RE: [ceph-users] cephfs causing high load on vm, taking down 15 
min later another cephfs vm


I have got this today again? I cannot unmount the filesystem and 
looks like some osd's are having 100% cpu utilization?


-Original Message-
From: Marc Roos
Sent: maandag 20 mei 2019 12:42
To: ceph-users
Subject: [ceph-users] cephfs causing high load on vm, taking down 15 min 
later another cephfs vm



I got my first problem with cephfs in a production environment. Is it 
possible from these logfiles to deduct what happened?

svr1 is connected to ceph client network via switch
svr2 vm is collocated on c01 node.
c01 has osd's and the mon.a colocated. 

svr1 was the first to report errors at 03:38:44. I have no error 
messages reported of a network connection problem by any of the ceph 
nodes. I have nothing in dmesg on c01.

[@c01 ~]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
[@c01 ~]# uname -a
Linux c01 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019 

x86_64 x86_64 x86_64 GNU/Linux
[@c01 ~]# ceph versions
{
"mon": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 

luminous (stable)": 3
},
"mgr": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 

luminous (stable)": 3
},
"osd": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 

luminous (stable)": 32
},
"mds": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 

luminous (stable)": 2
},
"rgw": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 

luminous (stable)": 2
},
"overall": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 

luminous (stable)": 42
}
}




[0] svr1 messages 
May 20 03:36:01 svr1 systemd: Started Session 308978 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308979 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308979 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308980 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308980 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308981 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308981 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308982 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308982 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308983 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308983 of user root.
May 20 03:38:44 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:44 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: last message repeated 5 times
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: last message repeated 5 times
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:

Re: [ceph-users] cephfs causing high load on vm, taking down 15 min later another cephfs vm

[Marc Roos]

 

No, but even if, I never had any issues when running multiple scrubs.



-Original Message-
From: EDH - Manuel Rios Fernandez [mailto:mrios...@easydatahost.com] 
Sent: dinsdag 21 mei 2019 10:03
To: Marc Roos; 'ceph-users'
Subject: RE: [ceph-users] cephfs causing high load on vm, taking down 15 
min later another cephfs vm

Hi Marc

Is there any scrub / deepscrub running in the affected OSDs?

Best Regards,
Manuel

-Mensaje original-
De: ceph-users  En nombre de Marc 
Roos Enviado el: martes, 21 de mayo de 2019 10:01
Para: ceph-users ; Marc Roos 

Asunto: Re: [ceph-users] cephfs causing high load on vm, taking down 15 
min later another cephfs vm

 
I have evicted all client connections and have still high load on osd's 

And ceph osd pool stats shows still client activity?

pool fs_data id 20
  client io 565KiB/s rd, 120op/s rd, 0op/s wr




-Original Message-
From: Marc Roos
Sent: dinsdag 21 mei 2019 9:51
To: ceph-users@lists.ceph.com; Marc Roos
Subject: RE: [ceph-users] cephfs causing high load on vm, taking down 15 
min later another cephfs vm


I have got this today again? I cannot unmount the filesystem and 
looks like some osd's are having 100% cpu utilization?


-Original Message-
From: Marc Roos
Sent: maandag 20 mei 2019 12:42
To: ceph-users
Subject: [ceph-users] cephfs causing high load on vm, taking down 15 min 
later another cephfs vm



I got my first problem with cephfs in a production environment. Is it 
possible from these logfiles to deduct what happened?

svr1 is connected to ceph client network via switch
svr2 vm is collocated on c01 node.
c01 has osd's and the mon.a colocated. 

svr1 was the first to report errors at 03:38:44. I have no error 
messages reported of a network connection problem by any of the ceph 
nodes. I have nothing in dmesg on c01.

[@c01 ~]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
[@c01 ~]# uname -a
Linux c01 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019 


x86_64 x86_64 x86_64 GNU/Linux
[@c01 ~]# ceph versions
{
"mon": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 


luminous (stable)": 3
},
"mgr": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 


luminous (stable)": 3
},
"osd": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 


luminous (stable)": 32
},
"mds": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 


luminous (stable)": 2
},
"rgw": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 


luminous (stable)": 2
},
"overall": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 


luminous (stable)": 42
}
}




[0] svr1 messages 
May 20 03:36:01 svr1 systemd: Started Session 308978 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308979 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308979 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308980 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308980 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308981 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308981 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308982 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308982 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308983 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308983 of user root.
May 20 03:38:44 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:44 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: last message repeated 5 times
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: last message repeated 5 times
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
established
May 20 03:38:

Re: [ceph-users] cephfs causing high load on vm, taking down 15 min later another cephfs vm

[Marc Roos]

 
I have evicted all client connections and have still high load on osd's 

And ceph osd pool stats shows still client activity?

pool fs_data id 20
  client io 565KiB/s rd, 120op/s rd, 0op/s wr




-Original Message-
From: Marc Roos 
Sent: dinsdag 21 mei 2019 9:51
To: ceph-users@lists.ceph.com; Marc Roos
Subject: RE: [ceph-users] cephfs causing high load on vm, taking down 15 
min later another cephfs vm


I have got this today again? I cannot unmount the filesystem and 
looks like some osd's are having 100% cpu utilization?


-Original Message-
From: Marc Roos
Sent: maandag 20 mei 2019 12:42
To: ceph-users
Subject: [ceph-users] cephfs causing high load on vm, taking down 15 min 
later another cephfs vm



I got my first problem with cephfs in a production environment. Is it 
possible from these logfiles to deduct what happened?

svr1 is connected to ceph client network via switch
svr2 vm is collocated on c01 node.
c01 has osd's and the mon.a colocated. 

svr1 was the first to report errors at 03:38:44. I have no error 
messages reported of a network connection problem by any of the ceph 
nodes. I have nothing in dmesg on c01.

[@c01 ~]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
[@c01 ~]# uname -a
Linux c01 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019 

x86_64 x86_64 x86_64 GNU/Linux
[@c01 ~]# ceph versions
{
"mon": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 

luminous (stable)": 3
},
"mgr": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 

luminous (stable)": 3
},
"osd": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 

luminous (stable)": 32
},
"mds": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 

luminous (stable)": 2
},
"rgw": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 

luminous (stable)": 2
},
"overall": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 

luminous (stable)": 42
}
}




[0] svr1 messages 
May 20 03:36:01 svr1 systemd: Started Session 308978 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308979 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308979 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308980 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308980 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308981 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308981 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308982 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308982 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308983 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308983 of user root.
May 20 03:38:44 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:44 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: last message repeated 5 times
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: last message repeated 5 times
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon2 192.168.x.113:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: mon2 192.168.x.113:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: libceph:

Re: [ceph-users] cephfs causing high load on vm, taking down 15 min later another cephfs vm

[Marc Roos]

I have got this today again? I cannot unmount the filesystem and 
looks like some osd's are having 100% cpu utilization?


-Original Message-
From: Marc Roos 
Sent: maandag 20 mei 2019 12:42
To: ceph-users
Subject: [ceph-users] cephfs causing high load on vm, taking down 15 min 
later another cephfs vm



I got my first problem with cephfs in a production environment. Is it 
possible from these logfiles to deduct what happened?

svr1 is connected to ceph client network via switch
svr2 vm is collocated on c01 node.
c01 has osd's and the mon.a colocated. 

svr1 was the first to report errors at 03:38:44. I have no error 
messages reported of a network connection problem by any of the ceph 
nodes. I have nothing in dmesg on c01.

[@c01 ~]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
[@c01 ~]# uname -a
Linux c01 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019 

x86_64 x86_64 x86_64 GNU/Linux
[@c01 ~]# ceph versions
{
"mon": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 

luminous (stable)": 3
},
"mgr": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 

luminous (stable)": 3
},
"osd": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 

luminous (stable)": 32
},
"mds": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 

luminous (stable)": 2
},
"rgw": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 

luminous (stable)": 2
},
"overall": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 

luminous (stable)": 42
}
}




[0] svr1 messages 
May 20 03:36:01 svr1 systemd: Started Session 308978 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308979 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308979 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308980 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308980 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308981 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308981 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308982 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308982 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308983 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308983 of user root.
May 20 03:38:44 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:44 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: last message repeated 5 times
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: last message repeated 5 times
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon2 192.168.x.113:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: mon2 192.168.x.113:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: libceph: mon2 192.168.x.113:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon2 192.168.x.113:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon2 192.168.x.113:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon2 192.168.x.113:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon0 192.16

[ceph-users] cephfs causing high load on vm, taking down 15 min later another cephfs vm

[Marc Roos]

I got my first problem with cephfs in a production environment. Is it 
possible from these logfiles to deduct what happened?

svr1 is connected to ceph client network via switch
svr2 vm is collocated on c01 node.
c01 has osd's and the mon.a colocated. 

svr1 was the first to report errors at 03:38:44. I have no error 
messages reported of a network connection problem by any of the ceph 
nodes. I have nothing in dmesg on c01.

[@c01 ~]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
[@c01 ~]# uname -a
Linux c01 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019 
x86_64 x86_64 x86_64 GNU/Linux
[@c01 ~]# ceph versions
{
"mon": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 
luminous (stable)": 3
},
"mgr": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 
luminous (stable)": 3
},
"osd": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 
luminous (stable)": 32
},
"mds": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 
luminous (stable)": 2
},
"rgw": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 
luminous (stable)": 2
},
"overall": {
"ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) 
luminous (stable)": 42
}
}




[0] svr1 messages 
May 20 03:36:01 svr1 systemd: Started Session 308978 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308979 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308979 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308980 of user root.
May 20 03:36:01 svr1 systemd: Started Session 308980 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308981 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308981 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308982 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308982 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308983 of user root.
May 20 03:38:01 svr1 systemd: Started Session 308983 of user root.
May 20 03:38:44 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:44 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: last message repeated 5 times
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: last message repeated 5 times
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon2 192.168.x.113:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: mon2 192.168.x.113:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error
May 20 03:38:45 svr1 kernel: libceph: mon2 192.168.x.113:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon2 192.168.x.113:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon2 192.168.x.113:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon2 192.168.x.113:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
established
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
lost, hunting for new mon
May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session 
established


[1] svr2 messages 
May 20 03:40:01 svr2 systemd: Stopping User Slice of root.
May 20 03:40:01 svr2 systemd: Removed slice User Slice of root.
May 20 03:40:01 

Re: [ceph-users] Default min_size value for EC pools

[Marc Roos]

 

https://ceph.com/community/new-luminous-erasure-coding-rbd-cephfs/


-Original Message-
From: Florent B [mailto:flor...@coppint.com] 
Sent: zondag 19 mei 2019 12:06
To: Paul Emmerich
Cc: Ceph Users
Subject: Re: [ceph-users] Default min_size value for EC pools

Thank you Paul for your answer. On a recent setup on Luminous I had by 
default min_size=k+m with k=2 and m=1.

When you say unsafe : what I would like to do with k=2 and m=1 is 
equivalent of replicated size=2 pool. In this context, is it really 
equivalent or EC pool is really unsafe ?

Thank you.


On 19/05/2019 11:41, Paul Emmerich wrote:


Default is k+1 or k if m == 1

min_size = k is unsafe and should never be set.


Paul

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at 
https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90


On Sun, May 19, 2019 at 11:31 AM Florent B  
wrote:


Hi,

I would like to know why default min_size value for EC pools 
is k+m ?

In this context, when a single OSD is down, the pool is 
unavailable (pgs
are "incomplete" and stuck queries start to grow).

Setting min_size=k seems the right setting, isn't it ?

Thank you.

Florent

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com






___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Poor performance for 512b aligned "partial" writes from Windows guests in OpenStack + potential fix

[Marc Roos]

Hmmm, looks like diskpart is of, reports the same about a volume, that 
fsutil fsinfo ntfsinfo c: report 512 (in this case correct, because it 
is on a ssd)
Anyone knows how to use fsutil with a path mounted disk (without drive 
letter)?


-Original Message-
From: Marc Roos 
Sent: donderdag 16 mei 2019 13:46
To: aderumier; trent.lloyd
Cc: ceph-users
Subject: Re: [ceph-users] Poor performance for 512b aligned "partial" 
writes from Windows guests in OpenStack + potential fix


I am not sure if it is possible to run fsutil on disk without drive 
letter, but mounted on path. 
So I used:
diskpart
select volume 3
Filesystems

And gives me this: 
Current File System

  Type : NTFS
  Allocation Unit Size : 4096
  Flags : 

File Systems Supported for Formatting

  Type : NTFS (Default)
  Allocation Unit Sizes: 512, 1024, 2048, 4096 (Default), 8192, 16K, 
32K, 64K

  Type : FAT32
  Allocation Unit Sizes: 4096, 8192 (Default), 16K, 32K, 64K

  Type : REFS
  Allocation Unit Sizes: 4096 (Default), 64K

So it looks like it detects 4k correctly? But I do not have the  in the disk of 
libvirt and have the WD with 512e:

[@c01 ~]# smartctl -a /dev/sdb | grep 'Sector Size'
Sector Sizes: 512 bytes logical, 4096 bytes physical

CentOS Linux release 7.6.1810 (Core)
ceph version 12.2.12
libvirt-4.5.0



-Original Message-
From: Trent Lloyd [mailto:trent.ll...@canonical.com]
Sent: donderdag 16 mei 2019 9:57
To: Alexandre DERUMIER
Cc: ceph-users
Subject: Re: [ceph-users] Poor performance for 512b aligned "partial" 
writes from Windows guests in OpenStack + potential fix

For libvirt VMs, first you need to add "" to the relevant 
 sections, and then stop/start the VM to apply the change.

Then you need to make sure your VirtIO drivers (the Fedora/Red Hat 
variety anyway) are from late 2018 or so. There was a bug fixed around 
July 2018, before that date, the physical_block_size=4096 parameter is 
not used by the Windows VirtIO driver (it was supposed to be, but did 
not work).

Relevant links:
https://bugzilla.redhat.com/show_bug.cgi?id=1428641
https://github.com/virtio-win/kvm-guest-drivers-windows/pull/312 

After that, you can check if Windows is correctly recognizing the 
physical block size,

Start cmd.exe with "Run as administrator", then run fsutil fsinfo 
ntfsinfo c:

It should show "Bytes Per Physical Sector : 4096"



Lastly at least for Windows itself this makes it do 4096-byte writes 
"most of the time", however some applications including Exchange have 
special handling of the sector size. I'm not really sure how MSSQL 
handles it, for example, it may or may not work correctly if you switch 
to 4096 bytes after installation - you may have to create new data files 
or something for it to do 4k segments - or not. Hopefully the MSSQL 
documentation has some information about that.

It is also possible to set logical_block_size=4096 as well as
physical_block_size=4096 ("4k native") however this absolutely causes 
problems with some software (e.g. exchange) if you convert an existing 
installation between the two. If you try to use 4k native mode, ideally 
you would want to do a fresh install, to avoid any such issues. Or 
again, refer to the docs and test it. Just beware it may cause issues if 
you try to switch to 4k native.

As a final note you can use this tool to process an OSD log with "debug 
filestore = 10" enabled, it will print out how many of the operations 
were unaligned:
https://github.com/lathiat/ceph-tools/blob/master/fstore_op_latency.rb


You can just enable debug filestore = 10 dynamically on 1 OSD for about
5 minutes, turn it off, and process the log. And you could compare 
before/after. I haven't written an equivalent tool for BlueStore 
unfortunately if you are already in the modern world :) I also didnt' 
check maybe debug osd or something also has the writes and offsets, so I 
could write a generic tool to cover both cases, but also I have not done 
that.



Hope that helps.

Regards,
Trent

On Thu, 16 May 2019 at 14:52, Alexandre DERUMIER 
wrote:


Many thanks for the analysis !


I'm going to test with 4K on heavy mssql database to see if I'm 
seeing improvement on ios/latency.
I'll report results in this thread.


- Mail original -
De: "Trent Lloyd" 
À: "ceph-users" 
Envoyé: Vendredi 10 Mai 2019 09:59:39
Objet: [ceph-users] Poor performance for 512b aligned "partial" 
writes from Windows guests in OpenStack + potential fix

I recently was investigating a performance problem for a reasonably 

sized OpenStack deployment having around 220 OSDs (3.5" 7200 RPM SAS 
HDD) with NVMe Journals. The primary workload is Windows guests backed 
by Cinder RBD volumes. 
Thi

Re: [ceph-users] Poor performance for 512b aligned "partial" writes from Windows guests in OpenStack + potential fix

[Marc Roos]

I am not sure if it is possible to run fsutil on disk without drive 
letter, but mounted on path. 
So I used:
diskpart
select volume 3
Filesystems

And gives me this: 
Current File System

  Type : NTFS
  Allocation Unit Size : 4096
  Flags : 

File Systems Supported for Formatting

  Type : NTFS (Default)
  Allocation Unit Sizes: 512, 1024, 2048, 4096 (Default), 8192, 16K, 
32K, 64K

  Type : FAT32
  Allocation Unit Sizes: 4096, 8192 (Default), 16K, 32K, 64K

  Type : REFS
  Allocation Unit Sizes: 4096 (Default), 64K

So it looks like it detects 4k correctly? But I do not have the  in the disk of 
libvirt and have the WD with 512e:

[@c01 ~]# smartctl -a /dev/sdb | grep 'Sector Size'
Sector Sizes: 512 bytes logical, 4096 bytes physical

CentOS Linux release 7.6.1810 (Core)
ceph version 12.2.12
libvirt-4.5.0



-Original Message-
From: Trent Lloyd [mailto:trent.ll...@canonical.com] 
Sent: donderdag 16 mei 2019 9:57
To: Alexandre DERUMIER
Cc: ceph-users
Subject: Re: [ceph-users] Poor performance for 512b aligned "partial" 
writes from Windows guests in OpenStack + potential fix

For libvirt VMs, first you need to add "" to the relevant 
 sections, and then stop/start the VM to apply the change.

Then you need to make sure your VirtIO drivers (the Fedora/Red Hat 
variety anyway) are from late 2018 or so. There was a bug fixed around 
July 2018, before that date, the physical_block_size=4096 parameter is 
not used by the Windows VirtIO driver (it was supposed to be, but did 
not work).

Relevant links:
https://bugzilla.redhat.com/show_bug.cgi?id=1428641
https://github.com/virtio-win/kvm-guest-drivers-windows/pull/312 

After that, you can check if Windows is correctly recognizing the 
physical block size,

Start cmd.exe with "Run as administrator", then run fsutil fsinfo 
ntfsinfo c:

It should show "Bytes Per Physical Sector : 4096"



Lastly at least for Windows itself this makes it do 4096-byte writes 
"most of the time", however some applications including Exchange have 
special handling of the sector size. I'm not really sure how MSSQL 
handles it, for example, it may or may not work correctly if you switch 
to 4096 bytes after installation - you may have to create new data files 
or something for it to do 4k segments - or not. Hopefully the MSSQL 
documentation has some information about that.

It is also possible to set logical_block_size=4096 as well as 
physical_block_size=4096 ("4k native") however this absolutely causes 
problems with some software (e.g. exchange) if you convert an existing 
installation between the two. If you try to use 4k native mode, ideally 
you would want to do a fresh install, to avoid any such issues. Or 
again, refer to the docs and test it. Just beware it may cause issues if 
you try to switch to 4k native.

As a final note you can use this tool to process an OSD log with "debug 
filestore = 10" enabled, it will print out how many of the operations 
were unaligned:
https://github.com/lathiat/ceph-tools/blob/master/fstore_op_latency.rb


You can just enable debug filestore = 10 dynamically on 1 OSD for about 
5 minutes, turn it off, and process the log. And you could compare 
before/after. I haven't written an equivalent tool for BlueStore 
unfortunately if you are already in the modern world :) I also didnt' 
check maybe debug osd or something also has the writes and offsets, so I 
could write a generic tool to cover both cases, but also I have not done 
that.



Hope that helps.

Regards,
Trent

On Thu, 16 May 2019 at 14:52, Alexandre DERUMIER  
wrote:


Many thanks for the analysis !


I'm going to test with 4K on heavy mssql database to see if I'm 
seeing improvement on ios/latency.
I'll report results in this thread.


- Mail original -
De: "Trent Lloyd" 
À: "ceph-users" 
Envoyé: Vendredi 10 Mai 2019 09:59:39
Objet: [ceph-users] Poor performance for 512b aligned "partial" 
writes from Windows guests in OpenStack + potential fix

I recently was investigating a performance problem for a reasonably 
sized OpenStack deployment having around 220 OSDs (3.5" 7200 RPM SAS 
HDD) with NVMe Journals. The primary workload is Windows guests backed 
by Cinder RBD volumes. 
This specific deployment is Ceph Jewel (FileStore + 
SimpleMessenger) which while it is EOL, the issue is reproducible on 
current versions and also on BlueStore however for different reasons 
than FileStore. 

Generally the Ceph cluster was suffering from very poor outlier 
performance, the numbers change a little bit depending on the exact 
situation but roughly 80% of I/O was happening in a "reasonable" time of 
0-200ms but 5-20% of I/O operations were taking excessively long 
anywhere from 500ms through to 10-20+ seconds. However the normal 
metrics for commit and apply latency were 

Re: [ceph-users] Huge rebalance after rebooting OSD host (Mimic)

[Marc Roos]

Are you sure your osd's are up and reachable? (run ceph osd tree on 
another node)



-Original Message-
From: Jan Kasprzak [mailto:k...@fi.muni.cz] 
Sent: woensdag 15 mei 2019 14:46
To: ceph-us...@ceph.com
Subject: [ceph-users] Huge rebalance after rebooting OSD host (Mimic)

Hello, Ceph users,

I wanted to install the recent kernel update on my OSD hosts with CentOS 
7, Ceph 13.2.5 Mimic. So I set a noout flag and ran "yum -y update" on 
the first OSD host. This host has 8 bluestore OSDs with data on HDDs and 
database on LVs of two SSDs (each SSD has 4 LVs for OSD metadata).

Everything went OK, so I rebooted this host. After the OSD host 
went back online, the cluster went from HEALTH_WARN (noout flag set) to 
HEALTH_ERR, and started to rebalance itself, with reportedly almost 60 % 
objects misplaced, and some of them degraded. And, of course 
backfill_toofull:

  cluster:
health: HEALTH_ERR
2300616/3975384 objects misplaced (57.872%)
Degraded data redundancy: 74263/3975384 objects degraded 
(1.868%), 146 pgs degraded, 122 pgs undersized
Degraded data redundancy (low space): 44 pgs 
backfill_toofull
 
  services:
mon: 3 daemons, quorum stratus1,stratus2,stratus3
mgr: stratus3(active), standbys: stratus1, stratus2
osd: 44 osds: 44 up, 44 in; 2022 remapped pgs
rgw: 1 daemon active
 
  data:
pools:   9 pools, 3360 pgs
objects: 1.33 M objects, 5.0 TiB
usage:   25 TiB used, 465 TiB / 490 TiB avail
pgs: 74263/3975384 objects degraded (1.868%)
 2300616/3975384 objects misplaced (57.872%)
 1739 active+remapped+backfill_wait
 1329 active+clean
 102  active+recovery_wait+remapped
 76   active+undersized+degraded+remapped+backfill_wait
 31   active+remapped+backfill_wait+backfill_toofull
 30   active+recovery_wait+undersized+degraded+remapped
 21   active+recovery_wait+degraded+remapped
 8
active+undersized+degraded+remapped+backfill_wait+backfill_toofull
 6active+recovery_wait+degraded
 4active+remapped+backfill_toofull
 3active+recovery_wait+undersized+degraded
 3active+remapped+backfilling
 2active+recovery_wait
 2active+recovering+undersized
 1active+clean+remapped
 1active+undersized+degraded+remapped+backfill_toofull
 1active+undersized+degraded+remapped+backfilling
 1active+recovering+undersized+remapped
 
  io:
client:   681 B/s rd, 1013 KiB/s wr, 0 op/s rd, 32 op/s wr
recovery: 142 MiB/s, 93 objects/s
 
(note that I cleaned the noout flag afterwards). What is wrong with it?
Why did the cluster decided to rebalance itself?

I am keeping the rest of the OSD hosts unrebooted for now.

Thanks,

-Yenya

-- 
| Jan "Yenya" Kasprzak  |
| http://www.fi.muni.cz/~kas/ GPG: 
4096R/A45477D5 |
sir_clive> I hope you don't mind if I steal some of your ideas?
 laryross> As far as stealing... we call it sharing here.   --from 
rcgroups
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Poor performance for 512b aligned "partial" writes from Windows guests in OpenStack + potential fix

[Marc Roos]

 
Hmmm, so if I have (wd) drives that list this in smartctl output, I 
should try and reformat them to 4k, which will give me better 
performance?

Sector Sizes: 512 bytes logical, 4096 bytes physical

Do you have a link to this download? Can only find some .cz site with 
the rpms. 


-Original Message-
From: Martin Verges [mailto:martin.ver...@croit.io] 
Sent: vrijdag 10 mei 2019 10:21
To: Trent Lloyd
Cc: ceph-users
Subject: Re: [ceph-users] Poor performance for 512b aligned "partial" 
writes from Windows guests in OpenStack + potential fix

Hello Trent,

many thanks for the insights. We always suggest to use 4kN over 512e 
HDDs to our users.

As we recently found out, is that WD Support offers a tool called HUGO 
to reformat 512e to 4kN drives with "hugo format -m  -n 
max --fastformat -b 4096" in seconds.
Maybe that helps someone that has bought the wrong disk.

--
Martin Verges
Managing director

Mobile: +49 174 9335695
E-Mail: martin.ver...@croit.io
Chat: https://t.me/MartinVerges

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht 
Munich HRB 231263

Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx



Am Fr., 10. Mai 2019 um 10:00 Uhr schrieb Trent Lloyd 
:


I recently was investigating a performance problem for a reasonably 
sized OpenStack deployment having around 220 OSDs (3.5" 7200 RPM SAS 
HDD) with NVMe Journals. The primary workload is Windows guests backed 
by Cinder RBD volumes.
This specific deployment is Ceph Jewel (FileStore + 
SimpleMessenger) which while it is EOL, the issue is reproducible on 
current versions and also on BlueStore however for different reasons 
than FileStore.


Generally the Ceph cluster was suffering from very poor outlier 
performance, the numbers change a little bit depending on the exact 
situation but roughly 80% of I/O was happening in a "reasonable" time of 
0-200ms but 5-20% of I/O operations were taking excessively long 
anywhere from 500ms through to 10-20+ seconds. However the normal 
metrics for commit and apply latency were normal, and in fact, this 
latency was hard to spot in the performance metrics available in jewel.

Previously I more simply considered FileStore to have the "commit" 
(to journal) stage where it was written to the journal and it is OK to 
return to the client and then the "apply" (to disk) stage where it was 
flushed to disk and confirmed so that the data could be purged from the 
journal. However there is really a third stage in the middle where 
FileStore submits the I/O to the operating system and this is done 
before the lock on the object is released. Until that succeeds another 
operation cannot write to the same object (generally being a 4MB area of 
the disk).

I found that the fstore_op threads would get stuck for hundreds of 
MS or more inside of pwritev() which was blocking inside of the kernel. 
Normally we expect pwritev() to be buffered I/O into the page cache and 
return quite fast however in this case the kernel was in a few percent 
of cases blocking with the stack trace included at the end of the e-mail 
[1]. My finding from that stack is that inside __block_write_begin_int 
we see a call to out_of_line_wait_on_bit call which is really an inlined 
call for wait_on_buffer which occurs in linux/fs/buffer.c in the section 
around line 2000-2024 with the comment "If we issued read requests - let 
them complete." 
(https://github.com/torvalds/linux/blob/a2d635decbfa9c1e4ae15cb05b68b255
9f7f827c/fs/buffer.c#L2002)

My interpretation of that code is that for Linux to store a write 
in the page cache, it has to have the entire 4K page as that is the 
granularity of which it tracks the dirty state and it needs the entire 
4K page to later submit back to the disk. Since we wrote a part of the 
page, and the page wasn't already in the cache, it has to fetch the 
remainder of the page from the disk. When this happens, it blocks 
waiting for this read to complete before returning from the pwritev() 
call - hence our normally buffered write blocks. This holds up the 
tp_fstore_op thread, of which there are (by default) only 2-4 such 
threads trying to process several hundred operations per second. 
Additionally the size of the osd_op_queue is bounded, and operations do 
not clear out of this queue until the tp_fstore_op thread is done. Which 
ultimately means that not only are these partial writes delayed but it 
knocks on to delay other writes behind them because of the constrained 
thread pools.

What was further confusing to this, is that I could easily 
reproduce this in a test deployment using an rbd benchmark that was only 
writing to a total disk size of 256MB which I would easily have expected 
to fit in the page cache:

rbd create -p rbd --size=256M bench2
rbd bench-write -p rbd bench2 --io-size 512 --io-threads 256 
--io-total 256M --io-pattern rand

  

Re: [ceph-users] maximum rebuild speed for erasure coding pool

[Marc Roos]

 
 > Fancy fast WAL/DB/Journals probably help a lot here, since they do 
affect the "iops"
 > you experience from your spin-drive OSDs.

What difference can be expected if you have a 100 iops hdd and you start 
using 
wal/db/journals on ssd? What would this 100 iops increase to 
(estimating)? 


-Original Message-
From: Janne Johansson [mailto:icepic...@gmail.com] 
Sent: donderdag 9 mei 2019 16:13
To: Feng Zhang
Cc: ceph-users
Subject: Re: [ceph-users] maximum rebuild speed for erasure coding pool

Den tors 9 maj 2019 kl 15:46 skrev Feng Zhang :



For erasure pool, suppose I have 10 nodes, each has 10 6TB drives, 
so
in total 100 drives. I make a 4+2 erasure pool, failure domain is
host/node. Then if one drive failed, (assume the 6TB is fully 
used),
what the maximum speed the recovering process can have? Also 
suppose
the cluster network is 10GbE, each disk has maximum 200MB/s 
sequential
throughput.




I think IOPS is what I experience makes the largest impact during 
recovery of spinning drives, not speed of sequential perf, so when 
recoverying you will see progress (on older clusters at least) in terms 
of number of misplaced/degraded objects like:

misplaced3454/34989583 objects (0.0123%)  

and the first number (here is my guess) moves at the speed of the IOPS 
of the drives being repaired, so if your drive(s) from which you rebuild 
can do 100 IOPS, then the above scenario will take ~34 seconds, even if 
that sizes and raw speeds should indicate something else about how fast 
they could move 0.0123% of your stored data.

As soon as you get super fast SSDs and NVMEs, the limit moves somewhere 
else since they have crazy IOPS numbers, and hence will repair lots 
faster, but if you have only spinning drives, then "don't hold your 
breath" is good advice, since it will take longer than 6 TB divided by 
200MB/s (8h20m) if you are unlucky and other drives can't help out in 
the rebuild.

Fancy fast WAL/DB/Journals probably help a lot here, since they do 
affect the "iops"
you experience from your spin-drive OSDs.

-- 

May the most significant bit of your life be positive.



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] EPEL packages issue

[Marc Roos]

It can only ask for the epel rego, if you have configured it somewhere, 
so remove it. There are some packages you need from epel. I have that 
mirrored, so I don't have such issues maybe you should mirror it to. 
Otherwise, these are the epel rpm's I have, you have to copy these then 
also to your custom repo. Or just see what the exact dependencies are. 

jemalloc.x86_64  3.6.0-1.el7 
@CentOS7_64-epel
leveldb.x86_64   1.12.0-11.el7   
@CentOS7_64-epel
libbabeltrace.x86_64 1.2.4-3.el7 
@CentOS7_64-epel
lttng-ust.x86_64 2.4.1-4.el7 
@CentOS7_64-epel
oniguruma.x86_64 5.9.5-3.el7 
@CentOS7_64-epel
perl-Net-SNMP.noarch 6.0.1-7.el7 
@CentOS7_64-epel
python-flask.noarch  1:0.10.1-3.el7  
@CentOS7_64-epel
python-pecan.noarch  0.4.5-2.el7 
@CentOS7_64-epel
python-simplegeneric.noarch  0.8-7.el7   
@CentOS7_64-epel
python-singledispatch.noarch 3.4.0.2-2.el7   
@CentOS7_64-epel
python-werkzeug.noarch   0.9.1-1.el7 
@CentOS7_64-epel
sockperf.x86_64  3.5-1.el7   
@CentOS7_64-epel
userspace-rcu.x86_64 0.7.16-1.el7
@CentOS7_64-epel






-Original Message-
From: Mohammad Almodallal [mailto:mmdal...@kku.edu.sa] 
Sent: dinsdag 7 mei 2019 16:18
To: Marc Roos; 'ceph-users'
Subject: RE: [ceph-users] EPEL packages issue

Hello, 

I already did this step and have the packages in local repostry, but 
still it aske for the EPEL repstry.

Regards.

mohammad almodallal


-Original Message-
From: Marc Roos 
Sent: Tuesday, May 7, 2019 10:15 AM
To: ceph-users ; mmdallal 

Subject: RE: [ceph-users] EPEL packages issue

 

I have the same situation, where the servers do not have internet 
connection and use my own repository servers. 

I am just rsyncing the rpms to my custom repository like this, works 
like a charm.

rsync -qrlptDvSHP --delete --exclude config.repo --exclude "local*" 
--exclude "aarch64" --exclude "drpms" --exclude "isos" 
anonym...@download.ceph.com::ceph/rpm-luminous/el7/
/var/www/cobbler/repo_mirror/Ceph-Luminous/



-Original Message-
From: Mohammad Almodallal [mailto:mmdal...@kku.edu.sa]
Sent: maandag 6 mei 2019 23:30
To: ceph-users@lists.ceph.com
Subject: [ceph-users] EPEL packages issue

Hello,

 

I need to install Ceph nautilus from local repository, I did download 
all the packages from Ceph site and created a local repository on the 
servers also servers don’t have internet access, but whenever I try to 
install Ceph it tries to install the EPEL release then the installation 
was failed.

 

Any help please?

 

Regards.

 

 

 





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] EPEL packages issue

[Marc Roos]

 

I have the same situation, where the servers do not have internet 
connection and use my own repository servers. 

I am just rsyncing the rpms to my custom repository like this, works 
like a charm.

rsync -qrlptDvSHP --delete --exclude config.repo --exclude "local*" 
--exclude "aarch64" --exclude "drpms" --exclude "isos" 
anonym...@download.ceph.com::ceph/rpm-luminous/el7/ 
/var/www/cobbler/repo_mirror/Ceph-Luminous/



-Original Message-
From: Mohammad Almodallal [mailto:mmdal...@kku.edu.sa] 
Sent: maandag 6 mei 2019 23:30
To: ceph-users@lists.ceph.com
Subject: [ceph-users] EPEL packages issue

Hello,

 

I need to install Ceph nautilus from local repository, I did download 
all the packages from Ceph site and created a local repository on the 
servers also servers don’t have internet access, but whenever I try to 
install Ceph it tries to install the EPEL release then the installation 
was failed.

 

Any help please?

 

Regards.

 

 

 


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph OSD fails to start : direct_read_unaligned error No data available

[Marc Roos]

The reason why you moved to ceph storage, is that you do not want to do such 
things. Remove the drive, and let ceph recover. 


On May 6, 2019 11:06 PM, Florent B wrote: > > Hi, > > It seems that OSD disk is 
dead (hardware problem), badblocks command > returns a lot of badblocks. > > Is 
there any way to force Ceph to recover as many objects as it can on > that disk 
? I know data will be lost but I would like to recover as many > as possible. > 
> Thank you. > > On 04/05/2019 18:56, Florent B wrote: > > Hi, > > > > I have a 
problem with an OSD that stopped itself and then can't restart. > > > > I use 
Luminous 12.2.12. > > > > I can see these lines in logs : > > > > -2> 
2019-05-04 18:48:33.687087 7f3aedfe1700  4 rocksdb: EVENT_LOG_v1 > > 
{"time_micros": 1556988513687079, "job": 3, "event": > > "compaction_started", 
"files_L1": [7456], "files_L2": [7414, 7415, 7416, > > 7417, 7418, 7419, 7420], 
"score": 1.25456, "input_data_size": 494084689} > >     -1> 2019-05-04 
18:48:33.947802 7f3b023bde00 -1 bdev(0x55ed83fb3200 > > 
/var/lib/ceph/osd/ceph-38/block) direct_read_unaligned > > 0x3cbb14c2b8~1c9631 
error: (61) No data available > >  0> 2019-05-04 18:48:33.952965 
7f3b023bde00 -1 > > /build/ceph-12.2.12/src/os/bluestore/BlueFS.cc: In function 
'int > > BlueFS::_read_random(BlueFS::FileReader*, uint64_t, size_t, char*)' > 
> thread 7f3b023bde00 time 2019-05-04 18:48:33.947917 > > 
/build/ceph-12.2.12/src/os/bluestore/BlueFS.cc: 936: FAILED assert(r == 0) > > 
> > Is there any way to recover this problem ? > > > > Thank you. > > > > 
Florent > > > > ___ > > ceph-users 
mailing list > > ceph-users@lists.ceph.com > > 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > 
___ > ceph-users mailing list > 
ceph-users@lists.ceph.com > 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd ssd pool for (windows) vms

[Marc Roos]

 
But what happens if the guest os has trim enabled and qemu did not have 
the discard option set. Should there be done some fsck to correct this? 
(Sorry is getting a bit off topic here.)


-Original Message-
From: Jason Dillaman [mailto:jdill...@redhat.com] 
Sent: woensdag 1 mei 2019 23:34
To: Marc Roos
Cc: ceph-users
Subject: Re: [ceph-users] rbd ssd pool for (windows) vms

On Wed, May 1, 2019 at 5:00 PM Marc Roos  
wrote:
>
>
> Do you need to tell the vm's that they are on a ssd rbd pool? Or does 
> ceph and the libvirt drivers do this automatically for you?

Like discard, any advanced QEMU options would need to be manually 
specified.

> When testing a nutanix acropolis virtual install, I had to 'cheat' it 
> by adding this  
> To make the installer think there was a ssd drive.
>
> I only have 'Thin provisioned drive' mentioned regardless if the vm is 

> on a hdd rbd pool or a ssd rbd pool.
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
Jason


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd ssd pool for (windows) vms

[Marc Roos]

 
Yes but those 'changes' can be relayed via the kernel rbd driver not? 
Besides I don't think you can move a rbd block device being used to a 
different pool anyway. 

On the manual page [0] there is nothing mentioned about configuration 
settings needed for rbd use. Nor for ssd. They are also using in the 
example the virtio/vda, while I learned here that you should use the 
virtio-scsi/sda.



[0] http://docs.ceph.com/docs/master/rbd/libvirt/





-Original Message-
From: Janne Johansson  
Subject: Re: [ceph-users] rbd ssd pool for (windows) vms

Den ons 1 maj 2019 kl 23:00 skrev Marc Roos :


Do you need to tell the vm's that they are on a ssd rbd pool? Or 
does 
ceph and the libvirt drivers do this automatically for you?
When testing a nutanix acropolis virtual install, I had to 'cheat' 
it by 
adding this
 
To make the installer think there was a ssd drive.  



Being or not being on an SSD pool is a (possibly) temporary conditition, 
so if the guest OS makes certain assumptions based on it, it might be 
invalid an hour later.




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] co-located cephfs client deadlock

[Marc Roos]

 
How did you retreive what osd nr to restart? 

Just for future reference, when I run into a similar situation. If you 
have a client hang on a osd node. This can be resolved by restarting
the osd that it is reading from?




-Original Message-
From: Dan van der Ster [mailto:d...@vanderster.com] 
Sent: donderdag 2 mei 2019 8:51
To: Yan, Zheng
Cc: ceph-users; pablo.llo...@cern.ch
Subject: Re: [ceph-users] co-located cephfs client deadlock

On Mon, Apr 1, 2019 at 1:46 PM Yan, Zheng  wrote:
>
> On Mon, Apr 1, 2019 at 6:45 PM Dan van der Ster  
wrote:
> >
> > Hi all,
> >
> > We have been benchmarking a hyperconverged cephfs cluster (kernel 
> > clients + osd on same machines) for awhile. Over the weekend (for 
> > the first time) we had one cephfs mount deadlock while some clients 
> > were running ior.
> >
> > All the ior processes are stuck in D state with this stack:
> >
> > [] wait_on_page_bit+0x83/0xa0 [] 

> > __filemap_fdatawait_range+0x111/0x190
> > [] filemap_fdatawait_range+0x14/0x30 
> > [] filemap_write_and_wait_range+0x56/0x90
> > [] ceph_fsync+0x55/0x420 [ceph] 
> > [] do_fsync+0x67/0xb0 [] 
> > SyS_fsync+0x10/0x20 [] 
> > system_call_fastpath+0x22/0x27 [] 
> > 0x
> >
>
> are there hang osd requests in /sys/kernel/debug/ceph/xxx/osdc?

We never managed to reproduce on this cluster.

But on a separate (not co-located) cluster we had a similar issue. A 
client was stuck like this for several hours:

HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs 
report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to 
respond to capability release
mdscephflax-mds-2a4cfd0e2c(mds.1): Client hpc070.cern.ch:hpcscid02 
failing to respond to capability release client_id: 69092525 
MDS_SLOW_REQUEST 1 MDSs report slow requests
mdscephflax-mds-2a4cfd0e2c(mds.1): 1 slow requests are blocked > 30 
sec


Indeed there was a hung write on hpc070.cern.ch:

245540  osd100  1.9443e2a5 1.2a5   [100,1,75]/100  [100,1,75]/100
e74658  
fsvolumens_393f2dcc-6b09-44d7-8d20-0e84b072ed26/2000b2f5905.0001
0x4000241 write

I restarted osd.100 and the deadlocked request went away.
Does this sound like a known issue?

Thanks, Dan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] hardware requirements for metadata server

[Marc Roos]

I have only 366M meta data stored in an ssd pool, with 16TB (10 million 
objects) of filesystem data (hdd pools).
The active mds is using 13GB memory.

Some stats of the active mds server
[@c01 ~]# ceph daemonperf mds.a
---mds --mds_cache--- --mds_log-- 
-mds_mem- mds_server- mds_ -objecter-- purg
req  rlat fwd  inos caps exi  imi |stry recy recd|subm evts segs 
repl|ino  dn  |hcr  hcs  hsr  cre |sess|actv rd   wr   rdwr|purg|
  000  3.6M  72k   00 | 28k   00 |  0   85k 1290 
|2.3M 3.6M|  0000 | 16 |  0000 |  0
  000  3.6M  72k   00 | 28k   00 |  0   85k 1290 
|2.3M 3.6M|  0000 | 16 |  0000 |  0
  000  3.6M  72k   00 | 28k   00 |  0   85k 1290 
|2.3M 3.6M|  0000 | 16 |  0000 |  0
  000  3.6M  72k   00 | 28k   00 |  0   85k 1290 
|2.3M 3.6M|  0000 | 16 |  0000 |  0
  000  3.6M  72k   00 | 28k   00 |  0   85k 1290 
|2.3M 3.6M|  0300 | 16 |  0000 |  0
  000  3.6M  72k   00 | 28k   00 |  6   85k 1290 
|2.3M 3.6M|  0500 | 16 |  0060 |  0
  000  3.6M  72k   00 | 28k   00 |  0   85k 1290 
|2.3M 3.6M|  0200 | 16 |  0000 |  0
  000  3.6M  72k   00 | 28k   00 |  0   85k 1290 
|2.3M 3.6M|  0000 | 16 |  0000 |  0
  000  3.6M  72k   00 | 28k   00 |  0   85k 1290 
|2.3M 3.6M|  0100 | 16 |  0000 |  0
  000  3.6M  72k   00 | 28k   00 |  3   85k 1290 
|2.3M 3.6M|  0100 | 16 |  0000 |  0

-Original Message-
From: Manuel Sopena Ballesteros [mailto:manuel...@garvan.org.au] 
Sent: donderdag 2 mei 2019 2:46
To: ceph-users@lists.ceph.com
Subject: [ceph-users] hardware requirements for metadata server

Dear Ceph users,

 

I would like to ask, does the metadata server needs much block devices 
for storage? Or does it only needs RAM? How could I calculate the amount 
of disks and/or memory needed?

 

Thank you very much

NOTICE
Please consider the environment before printing this email. This message 
and any attachments are intended for the addressee named and may contain 
legally privileged/confidential/copyright information. If you are not 
the intended recipient, you should not read, use, disclose, copy or 
distribute this communication. If you have received this message in 
error please notify us at once by return email and then delete both 
messages. We accept no liability for the distribution of viruses or 
similar in electronic communications. This notice should not be removed. 



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] rbd ssd pool for (windows) vms

[Marc Roos]

Do you need to tell the vm's that they are on a ssd rbd pool? Or does 
ceph and the libvirt drivers do this automatically for you?

When testing a nutanix acropolis virtual install, I had to 'cheat' it by 
adding this
 
To make the installer think there was a ssd drive. 

I only have 'Thin provisioned drive' mentioned regardless if the vm is 
on a hdd rbd pool or a ssd rbd pool.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph inside Docker containers inside VirtualBox

[Marc Roos]

 

I am not sure about your background knowledge of ceph, but if you are 
starting. Maybe first try and get ceph working in a virtual environment, 
that should not be to much of a problem. Then try migrating it to your 
container. Now you are probably fighting to many issues at the same 
time.  





-Original Message-
From: Varun Singh [mailto:varun.si...@gslab.com] 
Sent: 22 April 2019 07:46
To: ceph-us...@ceph.com
Subject: Re: [ceph-users] Ceph inside Docker containers inside 
VirtualBox

On Fri, Apr 19, 2019 at 6:53 PM Varun Singh  
wrote:
>
> On Fri, Apr 19, 2019 at 10:44 AM Varun Singh  
wrote:
> >
> > On Thu, Apr 18, 2019 at 9:53 PM Siegfried Höllrigl 
> >  wrote:
> > >
> > > Hi !
> > >
> > > I am not 100% sure, but i think, --net=host does not propagate 
> > > /dev/ inside the conatiner.
> > >
> > >  From the Error Message :
> > >
> > > 2019-04-18 07:30:06  /opt/ceph-container/bin/entrypoint.sh: ERROR- 

> > > The device pointed by OSD_DEVICE (/dev/vdd) doesn't exist !
> > >
> > >
> > > I whould say, you should add something like --device=/dev/vdd to 
the docker run command for the osd.
> > >
> > > Br
> > >
> > >
> > > Am 18.04.2019 um 14:46 schrieb Varun Singh:
> > > > Hi,
> > > > I am trying to setup Ceph through Docker inside a VM. My host 
> > > > machine is Mac. My VM is an Ubuntu 18.04. Docker version is 
> > > > 18.09.5, build e8ff056.
> > > > I am following the documentation present on ceph/daemon Docker 
> > > > Hub page. The idea is, if I spawn docker containers as mentioned 

> > > > on the page, I should get a ceph setup without KV store. I am 
> > > > not worried about KV store as I just want to try it out. 
> > > > Following are the commands I am firing to bring the containers 
up:
> > > >
> > > > Monitor:
> > > > docker run -d --net=host -v /etc/ceph:/etc/ceph -v 
> > > > /var/lib/ceph/:/var/lib/ceph/ -e MON_IP=10.0.2.15 -e
> > > > CEPH_PUBLIC_NETWORK=10.0.2.0/24 ceph/daemon mon
> > > >
> > > > Manager:
> > > > docker run -d --net=host -v /etc/ceph:/etc/ceph -v 
> > > > /var/lib/ceph/:/var/lib/ceph/ ceph/daemon mgr
> > > >
> > > > OSD:
> > > > docker run -d --net=host --pid=host --privileged=true -v 
> > > > /etc/ceph:/etc/ceph -v /var/lib/ceph/:/var/lib/ceph/ -v 
> > > > /dev/:/dev/ -e OSD_DEVICE=/dev/vdd ceph/daemon osd
> > > >
> > > >  From the above commands I am able to spawn monitor and manager 
> > > > properly. I verified this by firing this command on both monitor 

> > > > and manager containers:
> > > > sudo docker exec d1ab985 ceph -s
> > > >
> > > > I get following outputs for both:
> > > >
> > > >cluster:
> > > >  id: 14a6e40a-8e54-4851-a881-661a84b3441c
> > > >  health: HEALTH_OK
> > > >
> > > >services:
> > > >  mon: 1 daemons, quorum serverceph-VirtualBox (age 62m)
> > > >  mgr: serverceph-VirtualBox(active, since 56m)
> > > >  osd: 0 osds: 0 up, 0 in
> > > >
> > > >data:
> > > >  pools:   0 pools, 0 pgs
> > > >  objects: 0 objects, 0 B
> > > >  usage:   0 B used, 0 B / 0 B avail
> > > >  pgs:
> > > >
> > > > However when I try to bring up OSD using above command, it 
> > > > doesn't work. Docker logs show this output:
> > > > 2019-04-18 07:30:06  /opt/ceph-container/bin/entrypoint.sh: 
static:
> > > > does not generate config
> > > > 2019-04-18 07:30:06  /opt/ceph-container/bin/entrypoint.sh: 
> > > > ERROR- The device pointed by OSD_DEVICE (/dev/vdd) doesn't exist 
!
> > > >
> > > > I am not sure why the doc asks to pass /dev/vdd to OSD_DEVICE 
env var.
> > > > I know there are five different ways to spawning the OSD, but I 
> > > > am not able to figure out which one would be suitable for a 
> > > > simple deployment. If you could please let me know how to spawn 
> > > > OSDs using Docker, it would help a lot.
> > > >
> > > >
> >
> > Thanks Br, I will try this out today.
> >
> > --
> > Regards,
> > Varun Singh
>
> Hi,
> So following your suggestion I tried following two commands:
> 1. I added --device=/dev/vdd switch without removing OSD_DEVICE env 
> var. This resulted in same error before docker run -d --net=host 
> --pid=host --privileged=true --device=/dev/vdd -v /etc/ceph:/etc/ceph 
> -v /var/lib/ceph/:/var/lib/ceph/ -v /dev/:/dev/ -e OSD_DEVICE=/dev/vdd 

> ceph/daemon osd
>
>
> 2. Then I removed OSD_DEVICE env var and just added --device=/dev/vdd 
> switch docker run -d --net=host --pid=host --privileged=true 
> --device=/dev/vdd -v /etc/ceph:/etc/ceph -v 
> /var/lib/ceph/:/var/lib/ceph/ -v /dev/:/dev/  ceph/daemon osd
>
> OSD_DEVICE related error went away and I think ceph created an OSD 
> successfully. But it wasn't able to connect to cluster. Is it because 
> I did not give and network related information? I get the following 
> error now:
>
> 2019-04-18 08:30:47  /opt/ceph-container/bin/entrypoint.sh: static:
> does not generate config
> 2019-04-18 08:30:47  /opt/ceph-container/bin/entrypoint.sh:
> Bootstrapped OSD(s) found; using OSD directory
> 2019-04-18 08:30:47  

Re: [ceph-users] Osd update from 12.2.11 to 12.2.12

[Marc Roos]

 

I have only this in the default section, I think it is related to not 
having any configuration for some of these osd's. I 'forgot' to add the 
most recently added node [osd.x] sections. But in any case nothing afaik 
that should have them behave differently.

[osd]
osd journal size = 1024
osd pool default size = 3
osd pool default min size = 2
osd pool default pg num = 8
osd pool default pgp num = 8
# osd objectstore = bluestore
# osd max object size = 134217728
# osd max object size = 26843545600
osd scrub min interval = 172800

And these in the custom section

[osd.x]
public addr = 192.168.10.x
cluster addr = 10.0.0.x





-Original Message-
From: David Turner [mailto:drakonst...@gmail.com] 
Sent: 22 April 2019 22:34
To: Marc Roos
Cc: ceph-users
Subject: Re: [ceph-users] Osd update from 12.2.11 to 12.2.12

Do you perhaps have anything in the ceph.conf files on the servers with 
those OSDs that would attempt to tell the daemon that they are filestore 
osds instead of bluestore?  I'm sure you know that the second part [1] 
of the output in both cases only shows up after an OSD has been 
rebooted.  I'm sure this too could be cleaned up by adding that line to 
the ceph.conf file.

[1] rocksdb_separate_wal_dir = 'false' (not observed, change may require 
restart)

On Sun, Apr 21, 2019 at 8:32 AM  wrote:




Just updated luminous, and setting max_scrubs value back. Why do I 
get 
osd's reporting differently 


I get these:
osd.18: osd_max_scrubs = '1' (not observed, change may require 
restart) 
osd_objectstore = 'bluestore' (not observed, change may require 
restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may 
require 
restart)
osd.19: osd_max_scrubs = '1' (not observed, change may require 
restart) 
osd_objectstore = 'bluestore' (not observed, change may require 
restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may 
require 
restart)
osd.20: osd_max_scrubs = '1' (not observed, change may require 
restart) 
osd_objectstore = 'bluestore' (not observed, change may require 
restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may 
require 
restart)
osd.21: osd_max_scrubs = '1' (not observed, change may require 
restart) 
osd_objectstore = 'bluestore' (not observed, change may require 
restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may 
require 
restart)
osd.22: osd_max_scrubs = '1' (not observed, change may require 
restart) 
osd_objectstore = 'bluestore' (not observed, change may require 
restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may 
require 
restart)


And I get osd's reporting like this:
osd.23: osd_max_scrubs = '1' (not observed, change may require 
restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may 
require 
restart)
osd.24: osd_max_scrubs = '1' (not observed, change may require 
restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may 
require 
restart)
osd.25: osd_max_scrubs = '1' (not observed, change may require 
restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may 
require 
restart)
osd.26: osd_max_scrubs = '1' (not observed, change may require 
restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may 
require 
restart)
osd.27: osd_max_scrubs = '1' (not observed, change may require 
restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may 
require 
restart)
osd.28: osd_max_scrubs = '1' (not observed, change may require 
restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may 
require 
restart)







___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Are there any statistics available on how most production ceph clusters are being used?

[Marc Roos]

Double thanks for the on-topic reply. The other two repsonses, were 
making
me doubt if my chinese (which I didn't study) is better than my english.


 >> I am a bit curious on how production ceph clusters are being used. I 
am 
 >> reading here that the block storage is used a lot with openstack and 

 >> proxmox, and via iscsi with vmare. 
 >Have you looked at the Ceph User Surveys/Census?
 >https://ceph.com/ceph-blog/ceph-user-survey-2018-results/
 >https://ceph.com/geen-categorie/results-from-the-ceph-census/

Sort of what I was looking for, so 42% use rgw, of which 74% s3.
I guess this main archive usage, is mostly done by providers

 >> But I since nobody here is interested in a better rgw client for end 

 >> users. I am wondering if the rgw is even being used like this, and 
what 
 >> most production environments look like. 
 >Your end-user client thread was specifically asking targeting GUI
 >clients on OSX & Windows. I feel that the GUI client usage of S3
 >protocol has a much higher visibility to data size ratio than
 >automation/tooling usage.
 >
 >As the quantity of data by a single user increases, the odds that GUI
 >tools are used for it decreases, as it's MUCH more likely to be driven
 >by automation & tooling around the API.

Hmm, interesting. I am having more soho clients. And was thinking of
getting them such gui client.

 >My earliest Ceph production deployment was mostly RGW (~16TB raw), 
with
 >a little bit of RBD/iSCSI usage (~1TB of floating disk between VMs).
 >Very little of the RGW usage was GUI driven (there certainly was some,
 >because it made business sense to offer it rather than FTP sites; but 
it
 >tiny compared to the automation flows).
 >
 >My second production deployment I worked was Dreamhost's DreamObjects,
 >which was over 3PB then: and MOST of the usage was still not 
GUI-driven.
 >
 >I'm working at DigitalOcean's Spaces offering now; again, mostly 
non-GUI
 >access.
 >
 >For the second part of your original-query, I feel that any new 
clients
 >SHOULD not be RGW-specific; they should be able to work on a wide 
range
 >of services that expose the S3 API, and have a good test-suite around
 >that (s3-tests, but for testing the client implementation; even Boto 
is
 >not bug-free).
 >

I think if you take the perspective of some end user that associates s3,
with something like an audi and nothing else. It is quite necessary 
to have a client that is easy and secure to use, where you just enter
 preferably only two things, your access key and your secret.

The advantage of having a more rgw specific gui client, is that you
- do not have the default amazon 'advertisements' (think of storage 
classes etc.)
- less configuration options, everything ceph does not support we do not
  need to configure. 
- no ftp, no what ever else, just this s3
- you do not have configuration options that ceph doesn't offer 
  (eg. this life cycle, bucket access logging?)
  I can imagine if you have quite a few clients, you could get quite 
some
  questions to answer, about things not working.
- you have better support for specific things like multi tenant account, 
etc.
- for once the https urls are correctly advertised

Whether one likes it or not ceph is afaik not fully s3 compatible




 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Osd update from 12.2.11 to 12.2.12

[Marc Roos]

Just updated luminous, and setting max_scrubs value back. Why do I get 
osd's reporting differently 


I get these:
osd.18: osd_max_scrubs = '1' (not observed, change may require restart) 
osd_objectstore = 'bluestore' (not observed, change may require restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may require 
restart)
osd.19: osd_max_scrubs = '1' (not observed, change may require restart) 
osd_objectstore = 'bluestore' (not observed, change may require restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may require 
restart)
osd.20: osd_max_scrubs = '1' (not observed, change may require restart) 
osd_objectstore = 'bluestore' (not observed, change may require restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may require 
restart)
osd.21: osd_max_scrubs = '1' (not observed, change may require restart) 
osd_objectstore = 'bluestore' (not observed, change may require restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may require 
restart)
osd.22: osd_max_scrubs = '1' (not observed, change may require restart) 
osd_objectstore = 'bluestore' (not observed, change may require restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may require 
restart)


And I get osd's reporting like this:
osd.23: osd_max_scrubs = '1' (not observed, change may require restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may require 
restart)
osd.24: osd_max_scrubs = '1' (not observed, change may require restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may require 
restart)
osd.25: osd_max_scrubs = '1' (not observed, change may require restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may require 
restart)
osd.26: osd_max_scrubs = '1' (not observed, change may require restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may require 
restart)
osd.27: osd_max_scrubs = '1' (not observed, change may require restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may require 
restart)
osd.28: osd_max_scrubs = '1' (not observed, change may require restart) 
rocksdb_separate_wal_dir = 'false' (not observed, change may require 
restart)







___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Are there any statistics available on how most production ceph clusters are being used?

[Marc Roos]

I am a bit curious on how production ceph clusters are being used. I am 
reading here that the block storage is used a lot with openstack and 
proxmox, and via iscsi with vmare. 
But I since nobody here is interested in a better rgw client for end 
users. I am wondering if the rgw is even being used like this, and what 
most production environments look like. 

This could also be interesting information to decide in what direction 
ceph should develop in the future not?








___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] rgw windows/mac clients shitty, develop a new one?

[Marc Roos]

I have been looking a bit at the s3 clients available to be used, and I 
think they are quite shitty, especially this Cyberduck that processes 
files with default reading rights to everyone. I am in the process to 
advice clients to use for instance this mountain duck. But I am not to 
happy about it. I don't like the fact that everything has default 
settings for amazon or other stuff in there for ftp or what ever.

I am thinking of developing something in-house, more aimed at the ceph 
environments, easier/better to use. 

What I can think of:

- cheaper, free or maybe even opensource
- default settings for your ceph cluster
- only configuration for object storage (no amazon, rackspace, backblaze 
shit)
- default secure settings
- offer in the client only functionality that is available from the 
specific ceph release
- integration with the finder / explorer windows

I am curious who would be interested in a such new client? Maybe better 
to send me your wishes directly, and not clutter the mailing list with 
this.








___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Topology query

[Marc Roos]

 

AFAIK you at least risk with cephfs on osd nodes this 'kernel deadlock'? 
I have it also, but with enough memory. Search mailing list for this.
I am looking at similar setup, but with mesos and strugling with some 
cni plugin we have to develop.


-Original Message-
From: Bob Farrell [mailto:b...@homeflow.co.uk] 
Sent: donderdag 11 april 2019 20:45
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Topology query

Hello. I am running Ceph Nautilus v14.2.0 on Ubuntu Bionic 18.04 LTS.

I would like to ask if anybody could advise if there will be any 
potential problems with my setup as I am running a lot of services on 
each node.

I have 8 large dedicated servers, each with two physical disks. All 
servers run Docker Swarm and host numerous web applications.

I have also installed Ceph on each node (not in Docker). The secondary 
disk on each server hosts an LVM volume which is dedicated to Ceph. Each 
node runs one of each: osd, mon, mgr, mdss. I use CephFS to mount the 
data into each node's filesystem, which is then accessed by numerous 
containers via Docker bindmounts.

So far everything is working great but we haven't put anything under 
heavy load. I googled around to see if there are any potential problems 
with what I'm doing but couldn't find too much. There was one forum post 
I read [but can't find now] which warned against this unless using very 
latest glibc due to kernel fsync issues (IIRC) but this post was from 
2014 so I hope I'm safe ?

Thanks for the great project - I got this far just from reading the docs 
and writing my own Ansible script (wanted to learn Ceph properly). It's 
really good stuff. : )

Cheers,


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] VM management setup

[Marc Roos]

We have also hybrid ceph/libvirt-kvm setup, using some scripts to do 
live migration, do you have auto failover in your setup?



-Original Message-
From: jes...@krogh.cc [mailto:jes...@krogh.cc] 
Sent: 05 April 2019 21:34
To: ceph-users
Subject: [ceph-users] VM management setup

Hi. Knowing this is a bit off-topic but seeking recommendations and 
advise anyway.

We're seeking a "management" solution for VM's - currently in the 40-50 
VM - but would like to have better access in managing them and 
potintially migrate them across multiple hosts, setup block devices, 
etc, etc.

This is only to be used internally in a department where a bunch of 
engineering people will manage it, no costumers and that kind of thing.

Up until now we have been using virt-manager with kvm - and have been 
quite satisfied when we were in the "few vms", but it seems like the 
time to move on.

Thus we're looking for something "simple" that can help manage a 
ceph+kvm based setup -  the simpler and more to the point the better.

Any recommendations?

.. found a lot of names allready ..
OpenStack
CloudStack
Proxmox
..

But recommendations are truely welcome.

Thanks.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Recommended fs to use with rbd

[Marc Roos]

I would like to use rbd image from replicated hdd pool in a libvirt/kvm 
vm. 

1. What is the best filesystem to use with rbd, just standaard xfs?
2. Is there a recommended tuning for lvm on how to put multiple rbd 
images?






___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com