Re: [ceph-users] MDS: How to increase timeouts?
Hi, On 12/15/2015 10:22 PM, Gregory Farnum wrote: On Tue, Dec 15, 2015 at 10:21 AM, Burkhard Linke wrote: Hi, I have a setup with two MDS in active/standby configuration. During times of high network load / network congestion, the active MDS is bounced between both instances: 1. mons(?) decide that MDS A is crashed/not available due to missing heartbeats 2015-12-15 16:38:08.471608 7f880df10700 1 mds.beacon.ceph-storage-01 _send skipping beacon, heartbeat map not healthy 2015-12-15 16:38:10.534941 7f8813e4b700 1 heartbeat_map is_healthy 'MDS' had timed out after 15 ... 2015-12-15 16:38:15.468190 7f880e711700 1 heartbeat_map reset_timeout 'MDS' had timed out after 15 2015-12-15 16:38:17.734172 7f8811818700 1 mds.-1.-1 handle_mds_map i (192.168.6.129:6825/2846) dne in the mdsmap, respawning myself 2. Failover to standby MDS B 3. MDS B starts recover/rejoin (takes up to 15 minutes), introducing even more load 4. MDS A is respawned as new standby MDS 5. mons kick out MDS B after timeout 6. Failover to MDS A It takes 15 minutes to work through rejoin on your MDS? :/ You might try running your daemons in standby-replay instead of just standby, so that they have a warm cache. You could also try to figure out if the limiting factor is MDS throughput or OSD IOPs. The secondary MDS is configured for standby-replay. The OSDs and the network are probably the limiting factors. I'll try increasing mds_beacon_grace and the session timeout as John suggested. Thank you for helping, Burkhard ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Journal symlink broken / Ceph 0.94.5 / CentOS 6.7
Hi, A fresh server install on one of my nodes (and yum update) left me with CentOS 6.7 / Ceph 0.94.5. All the other nodes are running Ceph 0.94.2. "ceph-disk prepare /dev/sda /dev/sdc" seems to work as expected, but "ceph-disk activate / dev/sda1" fails. I have traced the problem to "/dev/disk/by-partuuid", where the journal symlinks are broken; -rw-r--r-- 1 root root 0 Dec 16 07:35 1e9d527f-0866-4284-b77c-c1cb04c5a168 -rw-r--r-- 1 root root 0 Dec 16 07:35 c34d4694-b486-450d-b57f-da24255f0072 lrwxrwxrwx 1 root root 10 Dec 16 07:35 c83b5aa5-fe77-42f6-9415-25ca0266fb7f -> ../../sdb1 lrwxrwxrwx 1 root root 10 Dec 16 07:35 e85f4d92-c8f1-4591-bd2a-aa43b80f58f6 -> ../../sda1 Re-creating them manually wont survive a reboot. Is this a problem with the udev rules in Ceph 0.94.3+? Hope that somebody can help me :-) Thanks! Best regards, Jesper ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] CentOS 7.2, Infernalis, preparing osd's and partprobe issues.
Hello,
On Wed, 16 Dec 2015 07:26:52 +0100 Mykola Dvornik wrote:
> I had more or less the same problem. This most likely synchronization
> issue. I have been deploying 16 OSD each running exactly the same
> hardware/software. The issue appeared randomly with no obvious
> correlations with other stuff. The dirty workaround was to put
> time.sleep(5) before invoking partprobe.
>
Yeah, I've been seeing this (or something eerily similar) as well with
Debian Wheezy and ceph 0.80.x, sometimes it would work, sometimes it would
need several tries.
I think this isn't helped by the udev magic, as it tries to get things
going before stuff is finished.
Christian
>
>
> On 16 December 2015 at 07:17, Matt Taylor wrote:
>
> > Hi all,
> >
> > After recently upgrading to CentOS 7.2 and installing a new Ceph
> > cluster using Infernalis v9.2.0, I have noticed that disk's are
> > failing to prepare.
> >
> > I have observed the same behaviour over multiple Ceph servers when
> > preparing disk's. All the servers are identical.
> >
> > Disk's are zapping fine, however when running 'ceph-deploy disk
> > prepare', we're encountering the following error:
> >
> > [ceph_deploy.cli][INFO ] Invoked (1.5.30): /usr/bin/ceph-deploy disk
> >> prepare kvsrv02:/dev/sdr
> >> [ceph_deploy.cli][INFO ] ceph-deploy options:
> >> [ceph_deploy.cli][INFO ] username : None
> >> [ceph_deploy.cli][INFO ] disk : [('kvsrv02', '/dev/sdr', None)]
> >> [ceph_deploy.cli][INFO ] dmcrypt : False
> >> [ceph_deploy.cli][INFO ] verbose : False
> >> [ceph_deploy.cli][INFO ] overwrite_conf : False
> >> [ceph_deploy.cli][INFO ] subcommand : prepare
> >> [ceph_deploy.cli][INFO ] dmcrypt_key_dir : /etc/ceph/dmcrypt-keys
> >> [ceph_deploy.cli][INFO ] quiet : False
> >> [ceph_deploy.cli][INFO ] cd_conf : >> instance at 0x7f1d54a4a7a0>
> >> [ceph_deploy.cli][INFO ] cluster : ceph
> >> [ceph_deploy.cli][INFO ] fs_type : xfs
> >> [ceph_deploy.cli][INFO ] func :
> >> [ceph_deploy.cli][INFO ] ceph_conf : None
> >> [ceph_deploy.cli][INFO ] default_release : False
> >> [ceph_deploy.cli][INFO ] zap_disk : False
> >> [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks
> >> kvsrv02:/dev/sdr: [kvsrv02][DEBUG ] connection detected need for sudo
> >> [kvsrv02][DEBUG ] connected to host: kvsrv02
> >> [kvsrv02][DEBUG ] detect platform information from remote host
> >> [kvsrv02][DEBUG ] detect machine type
> >> [kvsrv02][DEBUG ] find the location of an executable
> >> [ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.2.1511 Core
> >> [ceph_deploy.osd][DEBUG ] Deploying osd to kvsrv02
> >> [kvsrv02][DEBUG ] write cluster configuration
> >> to /etc/ceph/{cluster}.conf [ceph_deploy.osd][DEBUG ] Preparing host
> >> kvsrv02 disk /dev/sdr journal None activate False
> >> [kvsrv02][INFO ] Running command: sudo ceph-disk -v prepare --cluster
> >> ceph --fs-type xfs -- /dev/sdr
> >> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
> >> --check-allows-journal -i 0 --cluster ceph
> >> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
> >> --check-wants-journal -i 0 --cluster ceph
> >> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
> >> --check-needs-journal -i 0 --cluster ceph
> >> [kvsrv02][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdr uuid path is
> >> /sys/dev/block/65:16/dm/uuid
> >> [kvsrv02][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdr uuid path is
> >> /sys/dev/block/65:16/dm/uuid
> >> [kvsrv02][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdr uuid path is
> >> /sys/dev/block/65:16/dm/uuid
> >> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
> >> --cluster=ceph --show-config-value=fsid
> >> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
> >> --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
> >> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
> >> --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
> >> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
> >> --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
> >> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
> >> --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
> >> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
> >> --cluster=ceph --show-config-value=osd_journal_size
> >> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
> >> --cluster=ceph --name=osd. --lookup osd_cryptsetup_parameters
> >> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
> >> --cluster=ceph --name=osd. --lookup osd_dmcrypt_key_size
> >> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
> >> --cluster=ceph --name=osd. --lookup osd_dmcrypt_type
> >> [kvsrv02][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdr uuid path is
> >> /sys/dev/block/65:16/dm/uuid
> >> [kvsrv02][WARNIN] INFO:ceph-disk:Will colocate journal with data on
> >> /dev/sdr
> >> [kvsrv02][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev
Re: [ceph-users] CentOS 7.2, Infernalis, preparing osd's and partprobe issues.
I had more or less the same problem. This most likely synchronization
issue. I have been deploying 16 OSD each running exactly the same
hardware/software. The issue appeared randomly with no obvious correlations
with other stuff. The dirty workaround was to put time.sleep(5) before
invoking partprobe.
On 16 December 2015 at 07:17, Matt Taylor wrote:
> Hi all,
>
> After recently upgrading to CentOS 7.2 and installing a new Ceph cluster
> using Infernalis v9.2.0, I have noticed that disk's are failing to prepare.
>
> I have observed the same behaviour over multiple Ceph servers when
> preparing disk's. All the servers are identical.
>
> Disk's are zapping fine, however when running 'ceph-deploy disk prepare',
> we're encountering the following error:
>
> [ceph_deploy.cli][INFO ] Invoked (1.5.30): /usr/bin/ceph-deploy disk
>> prepare kvsrv02:/dev/sdr
>> [ceph_deploy.cli][INFO ] ceph-deploy options:
>> [ceph_deploy.cli][INFO ] username : None
>> [ceph_deploy.cli][INFO ] disk : [('kvsrv02', '/dev/sdr', None)]
>> [ceph_deploy.cli][INFO ] dmcrypt : False
>> [ceph_deploy.cli][INFO ] verbose : False
>> [ceph_deploy.cli][INFO ] overwrite_conf : False
>> [ceph_deploy.cli][INFO ] subcommand : prepare
>> [ceph_deploy.cli][INFO ] dmcrypt_key_dir : /etc/ceph/dmcrypt-keys
>> [ceph_deploy.cli][INFO ] quiet : False
>> [ceph_deploy.cli][INFO ] cd_conf : > instance at 0x7f1d54a4a7a0>
>> [ceph_deploy.cli][INFO ] cluster : ceph
>> [ceph_deploy.cli][INFO ] fs_type : xfs
>> [ceph_deploy.cli][INFO ] func :
>> [ceph_deploy.cli][INFO ] ceph_conf : None
>> [ceph_deploy.cli][INFO ] default_release : False
>> [ceph_deploy.cli][INFO ] zap_disk : False
>> [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks kvsrv02:/dev/sdr:
>> [kvsrv02][DEBUG ] connection detected need for sudo
>> [kvsrv02][DEBUG ] connected to host: kvsrv02
>> [kvsrv02][DEBUG ] detect platform information from remote host
>> [kvsrv02][DEBUG ] detect machine type
>> [kvsrv02][DEBUG ] find the location of an executable
>> [ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.2.1511 Core
>> [ceph_deploy.osd][DEBUG ] Deploying osd to kvsrv02
>> [kvsrv02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
>> [ceph_deploy.osd][DEBUG ] Preparing host kvsrv02 disk /dev/sdr journal
>> None activate False
>> [kvsrv02][INFO ] Running command: sudo ceph-disk -v prepare --cluster
>> ceph --fs-type xfs -- /dev/sdr
>> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
>> --check-allows-journal -i 0 --cluster ceph
>> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
>> --check-wants-journal -i 0 --cluster ceph
>> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
>> --check-needs-journal -i 0 --cluster ceph
>> [kvsrv02][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdr uuid path is
>> /sys/dev/block/65:16/dm/uuid
>> [kvsrv02][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdr uuid path is
>> /sys/dev/block/65:16/dm/uuid
>> [kvsrv02][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdr uuid path is
>> /sys/dev/block/65:16/dm/uuid
>> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
>> --cluster=ceph --show-config-value=fsid
>> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
>> --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
>> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
>> --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
>> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
>> --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
>> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
>> --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
>> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
>> --cluster=ceph --show-config-value=osd_journal_size
>> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
>> --cluster=ceph --name=osd. --lookup osd_cryptsetup_parameters
>> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
>> --cluster=ceph --name=osd. --lookup osd_dmcrypt_key_size
>> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
>> --cluster=ceph --name=osd. --lookup osd_dmcrypt_type
>> [kvsrv02][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdr uuid path is
>> /sys/dev/block/65:16/dm/uuid
>> [kvsrv02][WARNIN] INFO:ceph-disk:Will colocate journal with data on
>> /dev/sdr
>> [kvsrv02][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdr uuid path is
>> /sys/dev/block/65:16/dm/uuid
>> [kvsrv02][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdr uuid path is
>> /sys/dev/block/65:16/dm/uuid
>> [kvsrv02][WARNIN] DEBUG:ceph-disk:Creating journal partition num 2 size
>> 5120 on /dev/sdr
>> [kvsrv02][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk
>> --new=2:0:5120M --change-name=2:ceph journal
>> --partition-guid=2:7058473f-5c4a-4566-9a11-95cae71e5086
>> --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdr
>> [kvsrv02][DEBUG ] The operati
[ceph-users] CentOS 7.2, Infernalis, preparing osd's and partprobe issues.
Hi all,
After recently upgrading to CentOS 7.2 and installing a new Ceph cluster
using Infernalis v9.2.0, I have noticed that disk's are failing to prepare.
I have observed the same behaviour over multiple Ceph servers when
preparing disk's. All the servers are identical.
Disk's are zapping fine, however when running 'ceph-deploy disk
prepare', we're encountering the following error:
[ceph_deploy.cli][INFO ] Invoked (1.5.30): /usr/bin/ceph-deploy disk prepare
kvsrv02:/dev/sdr
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] disk : [('kvsrv02', '/dev/sdr', None)]
[ceph_deploy.cli][INFO ] dmcrypt : False
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : prepare
[ceph_deploy.cli][INFO ] dmcrypt_key_dir : /etc/ceph/dmcrypt-keys
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf :
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] fs_type : xfs
[ceph_deploy.cli][INFO ] func :
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] zap_disk : False
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks kvsrv02:/dev/sdr:
[kvsrv02][DEBUG ] connection detected need for sudo
[kvsrv02][DEBUG ] connected to host: kvsrv02
[kvsrv02][DEBUG ] detect platform information from remote host
[kvsrv02][DEBUG ] detect machine type
[kvsrv02][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.2.1511 Core
[ceph_deploy.osd][DEBUG ] Deploying osd to kvsrv02
[kvsrv02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.osd][DEBUG ] Preparing host kvsrv02 disk /dev/sdr journal None
activate False
[kvsrv02][INFO ] Running command: sudo ceph-disk -v prepare --cluster ceph
--fs-type xfs -- /dev/sdr
[kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
--check-allows-journal -i 0 --cluster ceph
[kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
--check-wants-journal -i 0 --cluster ceph
[kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
--check-needs-journal -i 0 --cluster ceph
[kvsrv02][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdr uuid path is
/sys/dev/block/65:16/dm/uuid
[kvsrv02][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdr uuid path is
/sys/dev/block/65:16/dm/uuid
[kvsrv02][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdr uuid path is
/sys/dev/block/65:16/dm/uuid
[kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
--cluster=ceph --show-config-value=fsid
[kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
--cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
[kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
--cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
[kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
--cluster=ceph --name=osd. --lookup osd_mount_options_xfs
[kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
--cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
[kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
--cluster=ceph --show-config-value=osd_journal_size
[kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
--cluster=ceph --name=osd. --lookup osd_cryptsetup_parameters
[kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
--cluster=ceph --name=osd. --lookup osd_dmcrypt_key_size
[kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
--cluster=ceph --name=osd. --lookup osd_dmcrypt_type
[kvsrv02][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdr uuid path is
/sys/dev/block/65:16/dm/uuid
[kvsrv02][WARNIN] INFO:ceph-disk:Will colocate journal with data on /dev/sdr
[kvsrv02][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdr uuid path is
/sys/dev/block/65:16/dm/uuid
[kvsrv02][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdr uuid path is
/sys/dev/block/65:16/dm/uuid
[kvsrv02][WARNIN] DEBUG:ceph-disk:Creating journal partition num 2 size 5120 on
/dev/sdr
[kvsrv02][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk --new=2:0:5120M
--change-name=2:ceph journal
--partition-guid=2:7058473f-5c4a-4566-9a11-95cae71e5086
--typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdr
[kvsrv02][DEBUG ] The operation has completed successfully.
[kvsrv02][WARNIN] DEBUG:ceph-disk:Calling partprobe on prepared device /dev/sdr
[kvsrv02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/udevadm settle
[kvsrv02][WARNIN] INFO:ceph-disk:Running command: /sbin/partprobe /dev/sdr
[kvsrv02][WARNIN] Error: Error informing the kernel about modifications to
partition /dev/sdr2 -- Device or resource busy. This means Linux won't know
about any changes you made to /dev/sdr2 until you reboot -- so you shouldn't
mount it or use it in any way before rebooting.
[kvsrv02][WARNIN] Error: Failed to add partition 2 (Device
Re: [ceph-users] MDS stuck replaying
On Tue, Dec 15, 2015 at 12:29 PM, Bryan Wright wrote: > John Spray writes: > >> If you haven't already, also >> check the overall health of the MDS host, e.g. is it low on >> memory/swapping? > > For what it's worth, I've taken down some OSDs, and that seems to have > allowed the MDS to finish replaying. My guess is that one of the OSDs was > having a problem that blocked the replay. Assuming this is so, is there any > way I could have seen that an OSD (and which one) was blocking the replay? The OSD was probably reporting slow ops via the central monitor log (ceph -w)? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] MDS: How to increase timeouts?
On Tue, Dec 15, 2015 at 10:21 AM, Burkhard Linke wrote: > Hi, > > I have a setup with two MDS in active/standby configuration. During times of > high network load / network congestion, the active MDS is bounced between > both instances: > > 1. mons(?) decide that MDS A is crashed/not available due to missing > heartbeats > > 2015-12-15 16:38:08.471608 7f880df10700 1 mds.beacon.ceph-storage-01 _send > skipping beacon, heartbeat map not healthy > 2015-12-15 16:38:10.534941 7f8813e4b700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > ... > 2015-12-15 16:38:15.468190 7f880e711700 1 heartbeat_map reset_timeout 'MDS' > had timed out after 15 > 2015-12-15 16:38:17.734172 7f8811818700 1 mds.-1.-1 handle_mds_map i > (192.168.6.129:6825/2846) dne in the mdsmap, respawning myself > > 2. Failover to standby MDS B > 3. MDS B starts recover/rejoin (takes up to 15 minutes), introducing even > more load > 4. MDS A is respawned as new standby MDS > 5. mons kick out MDS B after timeout > 6. Failover to MDS A It takes 15 minutes to work through rejoin on your MDS? :/ You might try running your daemons in standby-replay instead of just standby, so that they have a warm cache. You could also try to figure out if the limiting factor is MDS throughput or OSD IOPs. -Greg > > > I resolve this situation by shutting down one MDS completely and force the > cluster to use the remaining one: > 2015-12-15 16:40:20.618774 7f7ca6946700 1 mds.0.80 reconnect_done > 2015-12-15 16:40:25.815374 7f7ca6946700 1 mds.0.80 handle_mds_map i am now > mds.0.80 > 2015-12-15 16:40:25.815384 7f7ca6946700 1 mds.0.80 handle_mds_map state > change up:reconnect --> up:rejoin > 2015-12-15 16:40:25.815388 7f7ca6946700 1 mds.0.80 rejoin_start > 2015-12-15 16:40:43.159921 7f7ca894a700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > 2015-12-15 16:40:44.619252 7f7ca303e700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > 2015-12-15 16:40:44.619260 7f7ca303e700 1 mds.beacon.cb-dell-pe620r _send > skipping beacon, heartbeat map not healthy > 2015-12-15 16:40:48.160055 7f7ca894a700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > 2015-12-15 16:40:48.619315 7f7ca303e700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > 2015-12-15 16:40:48.619322 7f7ca303e700 1 mds.beacon.cb-dell-pe620r _send > skipping beacon, heartbeat map not healthy > 2015-12-15 16:40:52.619380 7f7ca303e700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > 2015-12-15 16:40:52.619405 7f7ca303e700 1 mds.beacon.cb-dell-pe620r _send > skipping beacon, heartbeat map not healthy > 2015-12-15 16:40:53.160157 7f7ca894a700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > 2015-12-15 16:40:56.619442 7f7ca303e700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > 2015-12-15 16:40:56.619452 7f7ca303e700 1 mds.beacon.cb-dell-pe620r _send > skipping beacon, heartbeat map not healthy > 2015-12-15 16:40:58.160257 7f7ca894a700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > 2015-12-15 16:41:00.619510 7f7ca303e700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > 2015-12-15 16:41:00.619519 7f7ca303e700 1 mds.beacon.cb-dell-pe620r _send > skipping beacon, heartbeat map not healthy > 2015-12-15 16:41:01.339416 7f7ca6946700 1 mds.0.80 rejoin_joint_start > 2015-12-15 16:41:01.343018 7f7ca383f700 1 heartbeat_map reset_timeout 'MDS' > had timed out after 15 > 2015-12-15 16:41:05.756154 7f7ca6946700 0 mds.beacon.cb-dell-pe620r > handle_mds_beacon no longer laggy > 2015-12-15 16:51:39.648189 7f7ca283d700 1 mds.0.80 rejoin_done > 2015-12-15 16:51:40.932766 7f7ca6946700 1 mds.0.80 handle_mds_map i am now > mds.0.80 > 2015-12-15 16:51:40.932783 7f7ca6946700 1 mds.0.80 handle_mds_map state > change up:rejoin --> up:active > 2015-12-15 16:51:40.932788 7f7ca6946700 1 mds.0.80 recovery_done -- > successful recovery! > 2015-12-15 16:51:43.235230 7f7ca6946700 1 mds.0.80 active_start > 2015-12-15 16:51:43.279305 7f7ca6946700 1 mds.0.80 cluster recovered. > ... > > How do I prevent the mons from removing the active MDS from the mdsmap or > allow a larger timeout? The documentation mentions mds_beacon_grace and > mds_beacon_interval, but it is not clear how these correlate to the > timeouts. > > How do I have to change the configuration to allow network congestions of up > to 5 minutes? > > Best regards, > Burkhard > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ACLs question in cephfs
On Tue, Dec 15, 2015 at 3:01 AM, Goncalo Borges wrote: > Dear Cephfs gurus. > > I have two questions regarding ACL support on cephfs. > > 1) Last time we tried ACLs we saw that they were only working properly in the > kernel module and I wonder what is the present status of acl support on > ceph-fuse. Can you clarify on that? Zheng has a PR pending integration testing for Jewel: https://github.com/ceph/ceph/pull/5658 It should go in master this week or next. > 2) If ceph-fuse is still not properly providing acl support is there a bad > consequences of using ceph-fuse in some hosts and kernel module in others, > setting ACLs in the latest ones? Presumably your permission checks won't make any sense? I don't know how it would really manifest, though; presumably it would depend on exactly what your ACLs are doing. -Greg > > TIA > Goncalo > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] MDS stuck replaying
John Spray writes: > If you haven't already, also > check the overall health of the MDS host, e.g. is it low on > memory/swapping? For what it's worth, I've taken down some OSDs, and that seems to have allowed the MDS to finish replaying. My guess is that one of the OSDs was having a problem that blocked the replay. Assuming this is so, is there any way I could have seen that an OSD (and which one) was blocking the replay? Bryan ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph Advisory Board Meeting
Hey cephers, In the interests of transparency, I wanted to share the resulting minutes from last week’s very first Ceph Advisory Board meeting: http://tracker.ceph.com/projects/ceph/wiki/CAB_2015-12-09 We are looking to meet monthly to discuss the following: * Pending development tasks for the next release * Current status of long-term projects * Coordination of efforts/resources to enable the Ceph community * Planning for future events * Handle incoming requests from the community * Anything else that may come up If you have questions about the form or function of the board, things that we discussed, or how you might request something from the Ceph Advisory Board please feel free to contact me. Thanks. -- Best Regards, Patrick McGarry Director Ceph Community || Red Hat http://ceph.com || http://community.redhat.com @scuttlemonkey || @ceph ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] MDS: How to increase timeouts?
On Tue, Dec 15, 2015 at 6:21 PM, Burkhard Linke wrote: > Hi, > > I have a setup with two MDS in active/standby configuration. During times of > high network load / network congestion, the active MDS is bounced between > both instances: > > 1. mons(?) decide that MDS A is crashed/not available due to missing > heartbeats > > 2015-12-15 16:38:08.471608 7f880df10700 1 mds.beacon.ceph-storage-01 _send > skipping beacon, heartbeat map not healthy > 2015-12-15 16:38:10.534941 7f8813e4b700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > ... > 2015-12-15 16:38:15.468190 7f880e711700 1 heartbeat_map reset_timeout 'MDS' > had timed out after 15 > 2015-12-15 16:38:17.734172 7f8811818700 1 mds.-1.-1 handle_mds_map i > (192.168.6.129:6825/2846) dne in the mdsmap, respawning myself > > 2. Failover to standby MDS B > 3. MDS B starts recover/rejoin (takes up to 15 minutes), introducing even > more load > 4. MDS A is respawned as new standby MDS > 5. mons kick out MDS B after timeout > 6. Failover to MDS A > > > I resolve this situation by shutting down one MDS completely and force the > cluster to use the remaining one: > 2015-12-15 16:40:20.618774 7f7ca6946700 1 mds.0.80 reconnect_done > 2015-12-15 16:40:25.815374 7f7ca6946700 1 mds.0.80 handle_mds_map i am now > mds.0.80 > 2015-12-15 16:40:25.815384 7f7ca6946700 1 mds.0.80 handle_mds_map state > change up:reconnect --> up:rejoin > 2015-12-15 16:40:25.815388 7f7ca6946700 1 mds.0.80 rejoin_start > 2015-12-15 16:40:43.159921 7f7ca894a700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > 2015-12-15 16:40:44.619252 7f7ca303e700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > 2015-12-15 16:40:44.619260 7f7ca303e700 1 mds.beacon.cb-dell-pe620r _send > skipping beacon, heartbeat map not healthy > 2015-12-15 16:40:48.160055 7f7ca894a700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > 2015-12-15 16:40:48.619315 7f7ca303e700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > 2015-12-15 16:40:48.619322 7f7ca303e700 1 mds.beacon.cb-dell-pe620r _send > skipping beacon, heartbeat map not healthy > 2015-12-15 16:40:52.619380 7f7ca303e700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > 2015-12-15 16:40:52.619405 7f7ca303e700 1 mds.beacon.cb-dell-pe620r _send > skipping beacon, heartbeat map not healthy > 2015-12-15 16:40:53.160157 7f7ca894a700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > 2015-12-15 16:40:56.619442 7f7ca303e700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > 2015-12-15 16:40:56.619452 7f7ca303e700 1 mds.beacon.cb-dell-pe620r _send > skipping beacon, heartbeat map not healthy > 2015-12-15 16:40:58.160257 7f7ca894a700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > 2015-12-15 16:41:00.619510 7f7ca303e700 1 heartbeat_map is_healthy 'MDS' > had timed out after 15 > 2015-12-15 16:41:00.619519 7f7ca303e700 1 mds.beacon.cb-dell-pe620r _send > skipping beacon, heartbeat map not healthy > 2015-12-15 16:41:01.339416 7f7ca6946700 1 mds.0.80 rejoin_joint_start > 2015-12-15 16:41:01.343018 7f7ca383f700 1 heartbeat_map reset_timeout 'MDS' > had timed out after 15 > 2015-12-15 16:41:05.756154 7f7ca6946700 0 mds.beacon.cb-dell-pe620r > handle_mds_beacon no longer laggy > 2015-12-15 16:51:39.648189 7f7ca283d700 1 mds.0.80 rejoin_done > 2015-12-15 16:51:40.932766 7f7ca6946700 1 mds.0.80 handle_mds_map i am now > mds.0.80 > 2015-12-15 16:51:40.932783 7f7ca6946700 1 mds.0.80 handle_mds_map state > change up:rejoin --> up:active > 2015-12-15 16:51:40.932788 7f7ca6946700 1 mds.0.80 recovery_done -- > successful recovery! > 2015-12-15 16:51:43.235230 7f7ca6946700 1 mds.0.80 active_start > 2015-12-15 16:51:43.279305 7f7ca6946700 1 mds.0.80 cluster recovered. > ... > > How do I prevent the mons from removing the active MDS from the mdsmap or > allow a larger timeout? The documentation mentions mds_beacon_grace and > mds_beacon_interval, but it is not clear how these correlate to the > timeouts. > > How do I have to change the configuration to allow network congestions of up > to 5 minutes? mds_beacon_grace is the timeout. A daemon is considered laggy if it has been more than mds_beacon_grace since it's last heartbeat was received by the mon. If you increase this, you'll have longer. You will probably also need to worry about mds_session_timeout, as the MDS also evicts client sessions that have not been in touch for longer than this. John > > Best regards, > Burkhard > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] MDS: How to increase timeouts?
Hi, I have a setup with two MDS in active/standby configuration. During times of high network load / network congestion, the active MDS is bounced between both instances: 1. mons(?) decide that MDS A is crashed/not available due to missing heartbeats 2015-12-15 16:38:08.471608 7f880df10700 1 mds.beacon.ceph-storage-01 _send skipping beacon, heartbeat map not healthy 2015-12-15 16:38:10.534941 7f8813e4b700 1 heartbeat_map is_healthy 'MDS' had timed out after 15 ... 2015-12-15 16:38:15.468190 7f880e711700 1 heartbeat_map reset_timeout 'MDS' had timed out after 15 2015-12-15 16:38:17.734172 7f8811818700 1 mds.-1.-1 handle_mds_map i (192.168.6.129:6825/2846) dne in the mdsmap, respawning myself 2. Failover to standby MDS B 3. MDS B starts recover/rejoin (takes up to 15 minutes), introducing even more load 4. MDS A is respawned as new standby MDS 5. mons kick out MDS B after timeout 6. Failover to MDS A I resolve this situation by shutting down one MDS completely and force the cluster to use the remaining one: 2015-12-15 16:40:20.618774 7f7ca6946700 1 mds.0.80 reconnect_done 2015-12-15 16:40:25.815374 7f7ca6946700 1 mds.0.80 handle_mds_map i am now mds.0.80 2015-12-15 16:40:25.815384 7f7ca6946700 1 mds.0.80 handle_mds_map state change up:reconnect --> up:rejoin 2015-12-15 16:40:25.815388 7f7ca6946700 1 mds.0.80 rejoin_start 2015-12-15 16:40:43.159921 7f7ca894a700 1 heartbeat_map is_healthy 'MDS' had timed out after 15 2015-12-15 16:40:44.619252 7f7ca303e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15 2015-12-15 16:40:44.619260 7f7ca303e700 1 mds.beacon.cb-dell-pe620r _send skipping beacon, heartbeat map not healthy 2015-12-15 16:40:48.160055 7f7ca894a700 1 heartbeat_map is_healthy 'MDS' had timed out after 15 2015-12-15 16:40:48.619315 7f7ca303e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15 2015-12-15 16:40:48.619322 7f7ca303e700 1 mds.beacon.cb-dell-pe620r _send skipping beacon, heartbeat map not healthy 2015-12-15 16:40:52.619380 7f7ca303e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15 2015-12-15 16:40:52.619405 7f7ca303e700 1 mds.beacon.cb-dell-pe620r _send skipping beacon, heartbeat map not healthy 2015-12-15 16:40:53.160157 7f7ca894a700 1 heartbeat_map is_healthy 'MDS' had timed out after 15 2015-12-15 16:40:56.619442 7f7ca303e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15 2015-12-15 16:40:56.619452 7f7ca303e700 1 mds.beacon.cb-dell-pe620r _send skipping beacon, heartbeat map not healthy 2015-12-15 16:40:58.160257 7f7ca894a700 1 heartbeat_map is_healthy 'MDS' had timed out after 15 2015-12-15 16:41:00.619510 7f7ca303e700 1 heartbeat_map is_healthy 'MDS' had timed out after 15 2015-12-15 16:41:00.619519 7f7ca303e700 1 mds.beacon.cb-dell-pe620r _send skipping beacon, heartbeat map not healthy 2015-12-15 16:41:01.339416 7f7ca6946700 1 mds.0.80 rejoin_joint_start 2015-12-15 16:41:01.343018 7f7ca383f700 1 heartbeat_map reset_timeout 'MDS' had timed out after 15 2015-12-15 16:41:05.756154 7f7ca6946700 0 mds.beacon.cb-dell-pe620r handle_mds_beacon no longer laggy 2015-12-15 16:51:39.648189 7f7ca283d700 1 mds.0.80 rejoin_done 2015-12-15 16:51:40.932766 7f7ca6946700 1 mds.0.80 handle_mds_map i am now mds.0.80 2015-12-15 16:51:40.932783 7f7ca6946700 1 mds.0.80 handle_mds_map state change up:rejoin --> up:active 2015-12-15 16:51:40.932788 7f7ca6946700 1 mds.0.80 recovery_done -- successful recovery! 2015-12-15 16:51:43.235230 7f7ca6946700 1 mds.0.80 active_start 2015-12-15 16:51:43.279305 7f7ca6946700 1 mds.0.80 cluster recovered. ... How do I prevent the mons from removing the active MDS from the mdsmap or allow a larger timeout? The documentation mentions mds_beacon_grace and mds_beacon_interval, but it is not clear how these correlate to the timeouts. How do I have to change the configuration to allow network congestions of up to 5 minutes? Best regards, Burkhard ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] MDS stuck replaying
John Spray writes: > Anyway -- you'll need to do some local poking of the MDS to work out > what the hold up is. Turn up MDS debug logging[1] and see what's > it's saying during the replay. Also, you can use performance counters > "ceph daemon mds. perf dump" and see which are incrementing to get > an idea of what it's doing. The "rd_pos" value from "perf dump > mds_log" should increment during replay. If you haven't already, also > check the overall health of the MDS host, e.g. is it low on > memory/swapping? > > John > > 1. http://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#runtime-changes > Hi John, Thanks for your help. There doesn't seem to be anything obvious wrong with any of the three MDSes. They're idle and have plenty of free memory and disk space. I briefly stopped the MDS that was active, and one of the others took over and resumed replaying. Looking at the "rd_pos" on the active MDS, it's sitting at a large number (226119121516) and hasn't changed in ten minutes or so. I've turned on debugging with "ceph tell mds.0 injectargs --debug-mds 20 --debug-ms 1 --debug-journaler=10", and here's a sample of what I see in the MDS log for the active MDS: 2015-12-15 13:21:55.507212 7fbe08aa2700 10 mds.beacon.2 _send up:replay seq 1525 2015-12-15 13:21:55.507238 7fbe08aa2700 1 -- 192.168.1.33:6800/13115 --> 192.168.1.33:6789/0 -- mdsbeacon(46356419/2 up:replay seq 1525 v78698) v3 -- ?+0 0x4e0f800 con 0x4118dc0 2015-12-15 13:21:55.508577 7fbe0c3aa700 1 -- 192.168.1.33:6800/13115 <== mon.2 192.168.1.33:6789/0 1592 mdsbeacon(46356419/2 up:replay seq 1525 v78698) v3 113+0+0 (2137559923 0 0) 0x7e06300 con 0x4118dc0 2015-12-15 13:21:55.508608 7fbe0c3aa700 10 mds.beacon.2 handle_mds_beacon up:replay seq 1525 rtt 0.001382 2015-12-15 13:21:59.484680 7fbe092a3700 10 MDSInternalContextBase::complete: N3MDS10C_MDS_TickE 2015-12-15 13:21:59.484740 7fbe092a3700 15 mds.0.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0.13> 2015-12-15 13:21:59.507303 7fbe08aa2700 10 mds.beacon.2 _send up:replay seq 1526 2015-12-15 13:21:59.507329 7fbe08aa2700 1 -- 192.168.1.33:6800/13115 --> 192.168.1.33:6789/0 -- mdsbeacon(46356419/2 up:replay seq 1526 v78698) v3 -- ?+0 0x4e0fb00 con 0x4118dc0 2015-12-15 13:21:59.508536 7fbe0c3aa700 1 -- 192.168.1.33:6800/13115 <== mon.2 192.168.1.33:6789/0 1593 mdsbeacon(46356419/2 up:replay seq 1526 v78698) v3 113+0+0 (4265335703 0 0) 0x7e06000 con 0x4118dc0 2015-12-15 13:21:59.508581 7fbe0c3aa700 10 mds.beacon.2 handle_mds_beacon up:replay seq 1526 rtt 0.001264 2015-12-15 13:21:59.510012 7fbe0cbab700 1 -- 192.168.1.33:6800/13115 --> 192.168.1.22:6812/5403 -- ping magic: 0 v1 -- ?+0 0x417e900 con 0x4c60580 2015-12-15 13:22:03.507394 7fbe08aa2700 10 mds.beacon.2 _send up:replay seq 1527 2015-12-15 13:22:03.507420 7fbe08aa2700 1 -- 192.168.1.33:6800/13115 --> 192.168.1.33:6789/0 -- mdsbeacon(46356419/2 up:replay seq 1527 v78698) v3 -- ?+0 0x4de8000 con 0x4118dc0 2015-12-15 13:22:03.508767 7fbe0c3aa700 1 -- 192.168.1.33:6800/13115 <== mon.2 192.168.1.33:6789/0 1594 mdsbeacon(46356419/2 up:replay seq 1527 v78698) v3 113+0+0 (2164974539 0 0) 0x7e05d00 con 0x4118dc0 2015-12-15 13:22:03.508799 7fbe0c3aa700 10 mds.beacon.2 handle_mds_beacon up:replay seq 1527 rtt 0.001390 2015-12-15 13:22:04.484781 7fbe092a3700 10 MDSInternalContextBase::complete: N3MDS10C_MDS_TickE 2015-12-15 13:22:04.484841 7fbe092a3700 15 mds.0.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0.12> 2015-12-15 13:22:04.510142 7fbe0cbab700 1 -- 192.168.1.33:6800/13115 --> 192.168.1.22:6812/5403 -- ping magic: 0 v1 -- ?+0 0x417eac0 con 0x4c60580 2015-12-15 13:22:07.507485 7fbe08aa2700 10 mds.beacon.2 _send up:replay seq 1528 2015-12-15 13:22:07.507511 7fbe08aa2700 1 -- 192.168.1.33:6800/13115 --> 192.168.1.33:6789/0 -- mdsbeacon(46356419/2 up:replay seq 1528 v78698) v3 -- ?+0 0x4de8300 con 0x4118dc0 2015-12-15 13:22:07.508757 7fbe0c3aa700 1 -- 192.168.1.33:6800/13115 <== mon.2 192.168.1.33:6789/0 1595 mdsbeacon(46356419/2 up:replay seq 1528 v78698) v3 113+0+0 (248276317 0 0) 0x7e05a00 con 0x4118dc0 2015-12-15 13:22:07.508788 7fbe0c3aa700 10 mds.beacon.2 handle_mds_beacon up:replay seq 1528 rtt 0.001288 2015-12-15 13:22:09.484881 7fbe092a3700 10 MDSInternalContextBase::complete: N3MDS10C_MDS_TickE 2015-12-15 13:22:09.484957 7fbe092a3700 15 mds.0.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0.11> 2015-12-15 13:22:09.510286 7fbe0cbab700 1 -- 192.168.1.33:6800/13115 --> 192.168.1.22:6812/5403 -- ping magic: 0 v1 -- ?+0 0x417ec80 con 0x4c60580 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ACLs question in cephfs
Dear Cephfs gurus. I have two questions regarding ACL support on cephfs. 1) Last time we tried ACLs we saw that they were only working properly in the kernel module and I wonder what is the present status of acl support on ceph-fuse. Can you clarify on that? 2) If ceph-fuse is still not properly providing acl support is there a bad consequences of using ceph-fuse in some hosts and kernel module in others, setting ACLs in the latest ones? TIA Goncalo ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] MDS stuck replaying
On Tue, Dec 15, 2015 at 5:01 PM, Bryan Wright wrote: > Hi folks, > > This morning, one of my MDSes dropped into "replaying": > > mds cluster is degraded > mds.0 at 192.168.1.31:6800/12550 rank 0 is replaying journal > > and the ceph filesystem seems to be unavailable to the clients. Is there > any way to see the progress of this replay? I don't see any indication in > the logs or elsewhere that it's actually doing anything. If it's safe to > truncate the journal, I'd be fine with just losing any changes made since > this morning, in order to get the filesystem back online. While you may see people dropping journals from time to time in disaster situations, be aware that it is not as simple as losing recent changes. Dropping the journal can easily leave you with an inconsistent filesystem (e.g. you appended to files, but updates to their size metadata are lost). I'm mainly mentioning this for the benefit of the list archive, as the topic of resetting journals comes up a fair bit. Anyway -- you'll need to do some local poking of the MDS to work out what the hold up is. Turn up MDS debug logging[1] and see what's it's saying during the replay. Also, you can use performance counters "ceph daemon mds. perf dump" and see which are incrementing to get an idea of what it's doing. The "rd_pos" value from "perf dump mds_log" should increment during replay. If you haven't already, also check the overall health of the MDS host, e.g. is it low on memory/swapping? John 1. http://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#runtime-changes ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] MDS stuck replaying
Hi folks, This morning, one of my MDSes dropped into "replaying": mds cluster is degraded mds.0 at 192.168.1.31:6800/12550 rank 0 is replaying journal and the ceph filesystem seems to be unavailable to the clients. Is there any way to see the progress of this replay? I don't see any indication in the logs or elsewhere that it's actually doing anything. If it's safe to truncate the journal, I'd be fine with just losing any changes made since this morning, in order to get the filesystem back online. The ceph version is 0.94.5, fwiw. Bryan ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] recommendations for file sharing
Currently, we use approach #1 with kerberized NFSv4 and Samba (with AD as KDC) - desperately waiting for CephFS :-) Best, Martin On Tue, Dec 15, 2015 at 11:51 AM, Wade Holler wrote: > Keep it simple is my approach. #1 > > If needed Add rudimentary HA with pacemaker. > > http://linux-ha.org/wiki/Samba > > Cheers > Wade > On Tue, Dec 15, 2015 at 5:45 AM Alex Leake wrote: > >> Good Morning, >> >> >> I have a production Ceph cluster at the University I work at, which runs >> brilliantly. >> >> >> However, I'd like your advice on the best way of sharing CIFS / SMB from >> Ceph. So far I have three ideas: >> >>1. Use a server as a head node, with an RBD mapped, then just >>export with samba >>2. Use a platform like OpenStack / KVM to host VMs that reside on >>Ceph RBDs - and use those to export filesystems (similar to 1.) >>3. Use a the S3 functionality of the RADOS gateway in conjunction >>with tools like SoftNAS (https://www.softnas.com/wp/) >> >> They all seem a little inefficient though, especially the 2nd option. >> >> Any help would be welcomed, I think it's an interesting problem. >> >> >> Kind Regards, >> Alex. >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] about federated gateway
Hi, I open an issue, and the link is as following: http://tracker.ceph.com/issues/14081 Thanks Sunfch 2015-12-15 2:33 GMT+08:00 Yehuda Sadeh-Weinraub : > On Sun, Dec 13, 2015 at 7:27 AM, 孙方臣 wrote: > > Hi, All, > > > > I'm setting up federated gateway. One is master zone, the other is slave > > zone. Radosgw-agent is running in slave zone. I have encountered some > > problems, can anybody help answering this: > > > > 1. When put a object to radosgw, there are two bilogs to generate. One > is > > "pending" state, the other is "complete" state.This should be ignored > when > > the entry is "pending" state, otherwise the same object will be copied > > twice. I have a pull request that is at > > https://github.com/ceph/radosgw-agent/pull/39, please give some > suggestions > > about it. > > > > 2. When the "rgw_num_rados_handles" is set as 16, the radosgw-agent > caannot > > unlock, the error code is 404. the log is following: > > .. > > 2015-12-13 21:52:33,373 26594 [radosgw_agent.lock][WARNING] failed to > unlock > > shard 115 in zone zone-a: Http error code 404 content Not Found > > .. > > 2015-12-13 21:53:00,732 26594 [radosgw_agent.lock][ERROR ] locking shard > 116 > > in zone zone-a failed: Http error code 423 content > > .. > > > > I can find the locker with the "rados lock info" command, and can break > the > > lock with "rados lock break" command. > > I find the reason finally, the reason is that the lock request from > > radosgw-agent is processed by rados client and the unlock request from > > radosgw-agent is processed by anther rados client. When the > > "rgw_num_rados_handles" is set as 1, the warning message did not > appeared. > > Can anybody help giving some suggestions about this, and can the warning > > message be ignored? > > Hi, > > it certainly seems like a bug. Can you open an issue at tracker.ceph.com? > > Thanks, > Yehuda > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Migrate Block Volumes and VMs
Hi everybody, My OpenStack System use Ceph as backend for Glance, Cinder, Nova. In the future, we intend build a new Ceph Cluster. I can re-connect current OpenStack with new Ceph systems. After that, I have tried export rbd images and import to new Ceph, but VMs and Volumes were clone of Glance rbd images, like this: rbd children images/e2c852e1-28ce-408d-b2ec-6351db35d55a@snap vms/8a4465fa-cbae-4559-b519-861eb4eda378_disk volumes/volume-b5937629-5f44-40c8-9f92-5f88129d3171 How could I export all rbd snapshot and its clones to import in new Ceph Cluster? Or is there any solution to move all Vms, Volumes, Images from old Ceph cluster to the new ones? Thanks and regards. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] recommendations for file sharing
Keep it simple is my approach. #1 If needed Add rudimentary HA with pacemaker. http://linux-ha.org/wiki/Samba Cheers Wade On Tue, Dec 15, 2015 at 5:45 AM Alex Leake wrote: > Good Morning, > > > I have a production Ceph cluster at the University I work at, which runs > brilliantly. > > > However, I'd like your advice on the best way of sharing CIFS / SMB from > Ceph. So far I have three ideas: > >1. Use a server as a head node, with an RBD mapped, then just export >with samba >2. Use a platform like OpenStack / KVM to host VMs that reside on Ceph >RBDs - and use those to export filesystems (similar to 1.) >3. Use a the S3 functionality of the RADOS gateway in conjunction with >tools like SoftNAS (https://www.softnas.com/wp/) > > They all seem a little inefficient though, especially the 2nd option. > > Any help would be welcomed, I think it's an interesting problem. > > > Kind Regards, > Alex. > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] recommendations for file sharing
On 12/15/2015 11:45 AM, Alex Leake wrote: > Good Morning, > > > I have a production Ceph cluster at the University I work at, which runs > brilliantly. > > > However, I'd like your advice on the best way of sharing CIFS / SMB from > Ceph. So far I have three ideas: > > 1. Use a server as a head node, with an RBD mapped, then just export > with samba > 2. Use a platform like OpenStack / KVM to host VMs that reside on Ceph > RBDs - and use those to export filesystems (similar to 1.) > 3. Use a the S3 functionality of the RADOS gateway in conjunction with > tools like SoftNAS (https://www.softnas.com/wp/) > > They all seem a little inefficient though, especially the 2nd option. > > Any help would be welcomed, I think it's an interesting problem. > Are you sure you need file sharing? ownCloud for example now has native RADOS support using phprados. Isn't ownCloud something that could work? Talking native RADOS is always the best. Wido > > Kind Regards, > Alex. > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Wido den Hollander 42on B.V. Ceph trainer and consultant Phone: +31 (0)20 700 9902 Skype: contact42on ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] recommendations for file sharing
Good Morning, I have a production Ceph cluster at the University I work at, which runs brilliantly. However, I'd like your advice on the best way of sharing CIFS / SMB from Ceph. So far I have three ideas: 1. ??Use a server as a head node, with an RBD mapped, then just export with samba 2. Use a platform like OpenStack / KVM to host VMs that reside on Ceph RBDs - and use those to export filesystems (similar to 1.) 3. Use a the S3 functionality of the RADOS gateway in conjunction with tools like SoftNAS (https://www.softnas.com/wp/)? They all seem a little inefficient though, especially the 2nd option. Any help would be welcomed, I think it's an interesting problem. Kind Regards, Alex. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
